Concepts
Streaming
Set "stream": true on /v1/chat/completions, /v1/responses or /v1/messages. The response is text/event-stream.
What is different about streams
- Headers come first.
x-request-id,x-utmost-model,x-utmost-route,x-utmost-fallbackand the rate-limit headers are sent. The token and cost headers are not, because nothing has been generated yet. Cost arrives in the final event. - No refusal retry. The router cannot retry a refusal it has already streamed to you. See Routing.
- No output safety check. For abliterated models, the gateway checks the input of every request, but checks the output only on non-streamed responses. See Safety.
- Fallback only before the first byte. If a backend fails before responding, the next one is tried. Once tokens are flowing, a failure ends the stream with an error event.
- Disconnects are billed. If you close the connection early, you are billed for the tokens generated up to that point.
Chat completions
Each event is a data: line holding a chat.completion.chunk. The stream ends with data: [DONE].
data: {"id":"chatcmpl-req_...","object":"chat.completion.chunk","created":1795066905,"model":"utmost/qwen3-30b-a3b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"id":"chatcmpl-req_...","object":"chat.completion.chunk","created":1795066905,"model":"utmost/qwen3-30b-a3b","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}
data: {"id":"chatcmpl-req_...","object":"chat.completion.chunk","created":1795066905,"model":"utmost/qwen3-30b-a3b","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]Deltas can carry content, reasoning_content and tool_calls. See Tool calling for tool deltas.
The usage chunk
Send "stream_options": {"include_usage": true} to get one more chunk before [DONE]. It has empty choices and the final usage, including cost_usd:
{
"id": "chatcmpl-req_...",
"object": "chat.completion.chunk",
"created": 1795066905,
"model": "utmost/qwen3-30b-a3b",
"choices": [],
"usage": {
"prompt_tokens": 5,
"completion_tokens": 22,
"total_tokens": 27,
"prompt_tokens_details": { "cached_tokens": 0 },
"completion_tokens_details": { "reasoning_tokens": 0 },
"cost_usd": 0.000007
}
}Without include_usage, a stream tells you nothing about its cost. The request is still metered, and the cost appears in the dashboard. The Utmost SDKs set include_usage by default.
Responses
Each event has an event: line naming its type and a data: line whose JSON repeats the type and carries a sequence_number. There is no [DONE] sentinel; the stream ends after response.completed or response.incomplete.
event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_...","status":"in_progress",...}}
event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":4,"item_id":"msg_...","output_index":0,"content_index":0,"delta":"Hello"}
event: response.completed
data: {"type":"response.completed","sequence_number":9,"response":{"id":"resp_...","status":"completed","usage":{...,"cost_usd":0.000007},...}}The full list of event types is on the Responses page. Usage, with cost_usd, is in the response object of the last event.
Messages (Anthropic)
The Anthropic event sequence:
| Event | Notes |
|---|---|
message_start |
usage here is all zeros |
ping |
Once, after message_start |
content_block_start |
A thinking, text or tool_use block |
content_block_delta |
thinking_delta, text_delta or input_json_delta |
content_block_stop |
|
message_delta |
stop_reason and the final usage |
message_stop |
The Anthropic usage object has no cost field. Read cost from the dashboard, or use chat completions or Responses when you need it per request.
Errors mid-stream
An error before the stream starts is an ordinary JSON error response with an HTTP status. An error after the stream starts arrives as a final event, after which the server closes the connection. The HTTP status is already 200 by then.
| Endpoint | Error event |
|---|---|
| Chat completions | data: {"error":{"message":"...","type":"api_error","code":"upstream_error","param":null,"request_id":"req_..."}}, with no [DONE] after it |
| Responses | event: error with data: {"type":"error","error_type":"api_error","message":"...","code":"upstream_error","param":null,"request_id":"req_...","sequence_number":N} |
| Messages | event: error with data: {"type":"error","error":{"type":"api_error","message":"...","code":"upstream_error"},"request_id":"req_..."} |
The usual mid-stream code is upstream_error: the backend's stream ended unexpectedly. Retrying is safe, but you are billed for the tokens the failed stream generated.
In the Responses error event, type is always error; the error's own type (for example api_error) is in error_type.
Reading a stream without an SDK
import json, os, urllib.request
req = urllib.request.Request(
"https://api.utmost.site/v1/chat/completions",
data=json.dumps({
"model": "auto", "stream": True, "stream_options": {"include_usage": True},
"messages": [{"role": "user", "content": "Say hello"}],
}).encode(),
headers={"Authorization": f"Bearer {os.environ['UTMOST_API_KEY']}", "Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as resp:
for raw in resp:
line = raw.decode().strip()
if not line.startswith("data: "):
continue
data = line[6:]
if data == "[DONE]":
break
chunk = json.loads(data)
if "error" in chunk:
raise RuntimeError(chunk["error"]["message"])
for choice in chunk["choices"]:
print(choice["delta"].get("content") or "", end="", flush=True)
if chunk.get("usage"):
print("\ncost:", chunk["usage"]["cost_usd"])