> For the complete documentation index, see [llms.txt](https://docs.kontinent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kontinent.ai/features/streaming.md).

# Streaming

Set `stream: true` on a chat completion to receive tokens as they are generated, over Server-Sent Events (`text/event-stream`). Every chat model in the catalog streams, including the ones Kontinent reaches through a non-OpenAI upstream (Claude on Bedrock and on Vertex): the gateway decodes their native formats and re-emits OpenAI-shaped chunks.

Embeddings have no incremental form: sending `stream: true` to `/v1/embeddings` is a `400`, not a silently ignored flag. A vector framed as an event stream is no use to anyone, and pretending to accept the parameter would leave you waiting for deltas that never come.

{% tabs %}
{% tab title="TypeScript" %}

```typescript
const stream = await client.chat.completions.create({
  model: "mistral/mistral-small-latest",
  messages: [{ role: "user", content: "Write a haiku about Europe." }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.error("\nusage:", chunk.usage);
}
```

{% endtab %}

{% tab title="Python" %}

```python
stream = client.chat.completions.create(
    model="mistral/mistral-small-latest",
    messages=[{"role": "user", "content": "Write a haiku about Europe."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print("\nusage:", chunk.usage)
```

{% endtab %}

{% tab title="curl" %}

```bash
curl -N https://api.kontinent.ai/v1/chat/completions \
  -H "Authorization: Bearer $KONTINENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"mistral/mistral-small-latest","stream":true,
       "messages":[{"role":"user","content":"Hallo!"}]}'
```

`-N` disables curl's own buffering. Without it you see the whole answer at once and learn nothing about streaming.
{% endtab %}
{% endtabs %}

## Wire format

Each event is a `data:` line carrying a JSON chunk, terminated by a sentinel:

```
data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-latest","choices":[{"index":0,"delta":{"role":"assistant","content":"Hal"},"finish_reason":null}]}

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-latest","choices":[{"index":0,"delta":{"content":"lo!"},"finish_reason":null}]}

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-latest","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-latest","choices":[],"usage":{"prompt_tokens":3,"completion_tokens":8,"total_tokens":11}}

data: [DONE]
```

Three things to rely on:

* The **first chunk usually announces the role** (`delta.role: "assistant"`), and never repeats it. Guaranteed on the chat paths Kontinent builds itself (Claude via Bedrock and Vertex); on passthrough providers the upstream's own chunks are forwarded untouched, and a few omit it. Treat the role as optional and default to `assistant`. (`/v1/responses` has no chat chunks and no `choices[].delta`: its named events carry their own `delta` payloads with the actual text, and the role arrives as `"role": "assistant"` on the message output item.)
* The **usage chunk has an empty `choices` array**. Kontinent injects `stream_options: {"include_usage": true}` upstream so it is always requested; if you set `stream_options` yourself, your value wins. When a provider omits usage anyway, the gateway estimates tokens — billing still happens, and the record is flagged internally as estimated.
* **`[DONE]` means the answer is complete.** It is the only positive signal that nothing is missing; see [Failure mid-stream](#failure-mid-stream) for the negative one.

### Keep-alive comments

While the model is still working, the gateway sends SSE comment lines so nothing in the path mistakes a thinking model for a dead connection:

```
: KONTINENT PROCESSING
```

A comment is not an event. Per the [SSE specification](https://html.spec.whatwg.org/multipage/server-sent-events.html#event-stream-interpretation) any line beginning with `:` carries no data, and every conforming client ignores it. You may use it to drive a loading indicator.

{% hint style="warning" %}
**If you parse the stream by hand, skip lines starting with `:` before calling `JSON.parse`.** Passing `: KONTINENT PROCESSING` to a JSON parser throws, and unhandled it crashes your read loop — a failure that only shows up on slow, expensive requests, because fast ones finish before the first keep-alive.
{% endhint %}

## Parsing the stream

The safest option is not to parse it yourself. These clients handle SSE framing, comments, multi-line `data:` fields and buffering correctly:

* the official **OpenAI SDKs** (point `baseURL` at `https://api.kontinent.ai/v1`)
* the **Vercel AI SDK**
* [**eventsource-parser**](https://github.com/rexxars/eventsource-parser), if you want the framing handled but the chunks yourself

If you do write the loop, it has to survive four things: a chunk split across two TCP reads, a comment line, the `[DONE]` sentinel, and an error chunk.

{% tabs %}
{% tab title="TypeScript" %}

```typescript
const res = await fetch("https://api.kontinent.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.KONTINENT_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "bedrock/claude-sonnet-5",
    messages: [{ role: "user", content: "Hallo!" }],
    stream: true,
  }),
});

// Failures BEFORE the first byte are plain JSON with a real status code.
if (!res.ok) {
  const { error } = await res.json();
  throw new Error(`${error.code}: ${error.message}`);
}
console.log("generation:", res.headers.get("X-Generation-Id"));

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";
let complete = false;

// A labelled loop, so `break read` can leave both loops from the inner one.
read: while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  let nl: number;
  while ((nl = buffer.indexOf("\n")) !== -1) {
    const line = buffer.slice(0, nl).trim();
    buffer = buffer.slice(nl + 1);

    if (line === "" || line.startsWith(":")) continue; // blank or keep-alive
    if (!line.startsWith("data:")) continue;

    const payload = line.slice(5).trim();
    if (payload === "[DONE]") { complete = true; break; }

    const chunk = JSON.parse(payload);

    // Failures AFTER the first byte arrive here; the status was already 200.
    if (chunk.error) {
      throw new Error(`${chunk.error.code}: ${chunk.error.message} (partial answer)`);
    }
    if (chunk.usage) console.error("usage:", chunk.usage);

    const delta = chunk.choices?.[0]?.delta;
    if (delta?.reasoning) process.stderr.write(delta.reasoning); // thinking
    if (delta?.content) process.stdout.write(delta.content);     // answer
  }
  if (complete) break read;
}

// EOF is not success. A connection that dropped before `[DONE]` leaves a
// truncated answer that looks exactly like a short one.
if (!complete) throw new Error("stream ended without [DONE]: answer is incomplete");
```

{% hint style="info" %}
This loop reads one `data:` line per event, which is all Kontinent sends. The SSE specification also permits several `data:` lines in one event, to be joined with newlines; if you want to be complete about that (or about `event:` and `id:` fields), use one of the parsers above rather than extending this loop.
{% endhint %}
{% endtab %}

{% tab title="Python" %}

```python
import json, requests

with requests.post(
    "https://api.kontinent.ai/v1/chat/completions",
    headers={"Authorization": f"Bearer {api_key}"},
    json={
        "model": "bedrock/claude-sonnet-5",
        "messages": [{"role": "user", "content": "Hallo!"}],
        "stream": True,
    },
    stream=True,
) as r:
    # Failures BEFORE the first byte are plain JSON with a real status code.
    if r.status_code != 200:
        err = r.json()["error"]
        raise RuntimeError(f"{err['code']}: {err['message']}")
    print("generation:", r.headers.get("X-Generation-Id"))

    complete = False
    for line in r.iter_lines(decode_unicode=True):
        if not line or line.startswith(":"):   # blank or keep-alive
            continue
        if not line.startswith("data:"):
            continue
        payload = line[5:].strip()
        if payload == "[DONE]":
            complete = True
            break
        chunk = json.loads(payload)

        # Failures AFTER the first byte arrive here; the status was already 200.
        if "error" in chunk:
            raise RuntimeError(f"{chunk['error']['code']}: {chunk['error']['message']} (partial answer)")
        if chunk.get("usage"):
            print("\nusage:", chunk["usage"])

        delta = (chunk.get("choices") or [{}])[0].get("delta", {})
        if delta.get("reasoning"):
            print(delta["reasoning"], end="", flush=True)   # thinking
        if delta.get("content"):
            print(delta["content"], end="", flush=True)     # answer

    # EOF is not success: without [DONE] the answer is truncated.
    if not complete:
        raise RuntimeError("stream ended without [DONE]: answer is incomplete")
```

{% endtab %}
{% endtabs %}

## Reasoning models

Thinking streams in the same `delta`, and it arrives **before** the answer:

```
data: {"choices":[{"index":0,"delta":{"role":"assistant","reasoning":"Comparing the ","reasoning_details":[{"type":"reasoning.text","text":"Comparing the ","index":0}]}}]}

data: {"choices":[{"index":0,"delta":{"reasoning":"tenths…","reasoning_details":[{"type":"reasoning.text","text":"tenths…","index":0}]}}]}

data: {"choices":[{"index":0,"delta":{"content":"9.9 is larger."}}]}
```

`delta.reasoning` is a string you can concatenate and print. `delta.reasoning_details` is the structured form you send back to continue a thinking conversation — see [Reasoning](/features/reasoning.md) for the request parameter, the detail types, and how encrypted thinking is reported.

{% hint style="info" %}
A reasoning model can be quiet for a long time before its first token — it is deliberating, not stuck. The gateway's keep-alive comments cover that gap, and on Bedrock and Vertex the thinking deltas themselves count as upstream activity.
{% endhint %}

## Tool calls

Tool calls stream as `delta.tool_calls`, on every chat model. The two Claude paths (Bedrock and Vertex) are translated into exactly this shape, so the wire format does not tell you which upstream served:

```
data: {"choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_a","type":"function","function":{"name":"get_weather","arguments":""}}]}}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"ci"}}]}}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ty\":\"Berlin\"}"}}]}}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}
```

Three rules, and they are OpenAI's:

* **`index` is the key, not the order of arrival.** The first fragment of a call carries `id`, `type` and `function.name`; every fragment after it carries argument text only. Two parallel calls interleave, so accumulate into a map keyed by `index`.
* **Arguments are a JSON string that arrives in pieces.** A single fragment is not parseable JSON. Concatenate first, `JSON.parse` once `finish_reason` has arrived.
* **`finish_reason: "tool_calls"`** is how the turn ends when the model wants a tool run. Anything else, including `"error"`, means the call you hold may be incomplete: never run a tool from a stream that did not finish.

To continue the loop, send the assistant turn back with its accumulated `tool_calls`, then one `{"role": "tool", "tool_call_id": "...", "content": "..."}` message per call. With a reasoning model, replay that assistant turn's `reasoning_details` too: Claude verifies its own thinking and refuses to continue from thinking it cannot attribute to itself. The official SDKs do the accumulation for you.

{% hint style="info" %}
Where an upstream cannot express a tool parameter, the request is refused with a `400` naming the field rather than served without it. On Claude via Bedrock that is `tool_choice: "none"` and `parallel_tool_calls: false`; on both Claude paths it is forcing a tool (`"required"` or a named function) while `reasoning` is enabled, which Anthropic rejects. See the [API reference](/reference/api-reference.md).
{% endhint %}

## Streaming `/v1/responses`

The Responses API streams **named** events rather than anonymous chunks, each carrying a monotonic `sequence_number`:

```
event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_01J…","status":"in_progress",…}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":1,"output_index":0,"item":{"type":"reasoning","id":"rs_01J…"}}

event: response.reasoning_text.delta
data: {"type":"response.reasoning_text.delta","sequence_number":2,"item_id":"rs_01J…","output_index":0,"content_index":0,"delta":"Comparing the "}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":7,"item_id":"msg_01J…","output_index":1,"content_index":0,"delta":"9.9 is larger."}

event: response.completed
data: {"type":"response.completed","sequence_number":11,"response":{"id":"resp_01J…","status":"completed","output":[…],"usage":{…}}}
```

The full event set is `response.created`, `response.output_item.added`, `response.reasoning_text.delta`, `response.reasoning_text.done`, `response.content_part.added`, `response.output_text.delta`, `response.output_text.done`, `response.content_part.done`, `response.function_call_arguments.delta`, `response.function_call_arguments.done`, `response.output_item.done`, and `response.completed`. Keep-alive comments and the `[DONE]`-free failure rule apply here too — a broken generation ends with `response.failed`, whose response object carries `status: "failed"`, an `error`, and whatever output was produced before the break.

A tool call is its own output item of type `function_call`, carrying `call_id`, `name` and `arguments`. Its arguments stream as `response.function_call_arguments.delta` fragments and are only parseable once `response.function_call_arguments.done` has repeated them whole. To continue the loop, send the item back in `input` as `{"type": "function_call", "call_id": "...", "name": "...", "arguments": "..."}` followed by `{"type": "function_call_output", "call_id": "...", "output": "..."}`.

Encrypted-only thinking produces **no** `response.reasoning_text.delta` — there is no plaintext to stream — but the blob still reaches you on the reasoning item of the final event.

See the [API reference](/reference/api-reference.md) for the request shape.

## Cancellation

Stop reading and close the connection — with `AbortController` in JavaScript, or by leaving the `with` block in Python. Kontinent closes the upstream connection in turn, so providers that honour cancellation stop generating immediately.

```typescript
const controller = new AbortController();
const stream = await client.chat.completions.create(
  { model: "mistral/mistral-small-latest", messages, stream: true },
  { signal: controller.signal },
);
// later:
controller.abort();
```

{% hint style="warning" %}
Cancelling **does not make the request free**. You are billed for the tokens generated up to that point. Because the provider never sends its usage chunk on a cancelled stream, that figure is estimated from what the gateway received.

Some upstreams do not honour cancellation at all (AWS Bedrock and Google among them) and finish the generation on their side regardless. Cancelling still stops the bytes reaching you; it does not always stop the work.
{% endhint %}

## Timeouts and keep-alives

| Limit                   | Value          | What it measures                                                                    |
| ----------------------- | -------------- | ----------------------------------------------------------------------------------- |
| Overall stream deadline | none           | A long answer is not a broken one.                                                  |
| Upstream idle           | **90 s**       | Silence from the *provider*. Its own chunks — including thinking deltas — reset it. |
| Keep-alive comment      | every **15 s** | Silence towards *you*. Proves the connection is alive.                              |
| Load-balancer idle      | 300 s          | Kontinent's own edge.                                                               |

The keep-alive deliberately does not reset the 90 s upstream limit. If it did, a provider that died mid-generation would hold the connection open indefinitely while the gateway kept reassuring you. The two answer different questions: *is the client still reachable* and *is the provider still working*.

When the 90 s limit does fire, you get the failure chunk described next — not a silent truncation.

## Failure mid-stream

If the model defines fallback providers, upstream `429`/`5xx`/connect errors are retried on the next provider — **only before the first byte** reaches you. That is the last moment a different provider can still serve the whole answer.

After the first byte the status code is already `200` and cannot be changed, so the failure is announced **inside the stream**:

```
data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"bedrock/claude-sonnet-5","provider":"bedrock","error":{"code":"upstream_error","message":"the provider failed mid-generation; the answer is incomplete"},"choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}
```

* The `error` object sits at the **top level**, beside the ordinary chunk fields.
* `finish_reason` is `"error"`, so a client that only tracks finish reasons still notices.
* `provider` names the upstream that dropped, which is what makes the failure actionable.
* **No `[DONE]` follows.** Detect the failure by testing each chunk for `error`, or for `finish_reason: "error"`.

Treat it as retriable on your side: the text you hold is partial, and the tokens already generated are billed. Errors that happen *before* streaming starts are ordinary JSON with a real status code (`400`, `401`, `402`, `429`, `502`, `503`) — see [Handling errors](/features/errors.md).

## Response headers

Every response — streaming or not, success or error — carries three headers:

| Header            | Meaning                                                   |
| ----------------- | --------------------------------------------------------- |
| `X-Generation-Id` | id of the usage record this request writes                |
| `X-Provider`      | the provider that actually served, **after** any failover |
| `X-Served-Model`  | the catalog model that actually served                    |

`X-Generation-Id` is the handle for reconciling a single call against your logs and your invoice. On the endpoints whose response body Kontinent builds itself (Bedrock, Vertex Anthropic, `/v1/responses`) the body's `id` is that value **with an endpoint prefix** — `chatcmpl-<id>` for chat completions, `resp_<id>` for `/v1/responses`. Strip the prefix before comparing; the two are not literally equal.

`X-Provider` and `X-Served-Model` are the only place you can see a silent failover. For a sovereignty product that is not a debugging nicety: you chose a model partly for *where* it runs, and after a fallback the answer may have come from your second choice.

## Checklist for a robust stream loop

1. Check the HTTP status before reading the body.
2. Record `X-Generation-Id` and `X-Provider`.
3. Buffer reads and split on newlines — a chunk can straddle two TCP reads.
4. Skip blank lines and lines starting with `:`.
5. Stop on `[DONE]`, and treat *any* other ending as incomplete.
6. Test each chunk for `error` / `finish_reason: "error"`.
7. Read `usage` from the chunk with empty `choices`.
8. Render `delta.reasoning` separately from `delta.content` — thoughts are not the answer.
9. Accumulate `delta.tool_calls` by `index`, concatenate `function.arguments`, and parse them only after `finish_reason: "tool_calls"`.

## Related

{% content-ref url="/pages/rr9HQO6ZIIKpSQKxpCW5" %}
[Reasoning](/features/reasoning.md)
{% endcontent-ref %}

{% content-ref url="/pages/e4XLAToYQziKWGuYMiZu" %}
[Handling errors](/features/errors.md)
{% endcontent-ref %}

{% content-ref url="/pages/AMjrYb6Jgzk2l2v9SVgU" %}
[Maximum availability](/models-and-routing/availability.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.kontinent.ai/features/streaming.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.