Streaming

Set stream: true on a chat completion to receive tokens as they are generated, over Server-Sent Events (text/event-stream).

Wire format

Each event is a data: line carrying a JSON chunk, terminated by a sentinel:

data: {"id":"chatcmpl-…","choices":[{"delta":{"content":"Hal"}}]}
data: {"id":"chatcmpl-…","choices":[{"delta":{"content":"lo!"}}]}
data: {"id":"chatcmpl-…","choices":[],"usage":{"prompt_tokens":3,"completion_tokens":8,"total_tokens":11}}
data: [DONE]

Kontinent injects stream_options: {"include_usage": true} upstream, so the final chunk before [DONE] carries usage. If a provider omits it, the gateway estimates tokens (billing still happens).

1
const stream = await client.chat.completions.create({
2
  model: "mistral/mistral-small-latest",
3
  messages: [{ role: "user", content: "Write a haiku about Europe." }],
4
  stream: true,
5
});
6
7
for await (const chunk of stream) {
8
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
9
}

Timeouts

• Streams have no overall deadline.
• An idle upstream is cut after 90 s of silence. The provider’s own chunks act as keep-alives.
• Behind the load balancer the idle timeout is 120 s.

Fallback and errors

If the model defines fallback providers, upstream 429/5xx/connect errors are retried on the next provider only before the first byte is streamed. Once the first byte is out, an error surfaces mid-stream as a dropped connection — handle a truncated stream gracefully.