Skip to content
Salyro
Get started
Guide

Stream responses

Show text as it is generated instead of waiting for the whole answer — enabling streaming, consuming it server-side, and handling a client that leaves.

Published

A model that takes six seconds to answer feels broken if nothing appears for six seconds, and feels fast if the first words arrive in three hundred milliseconds. Nothing about the model changes between those two experiences — only whether you asked for the answer in pieces.

This guide covers turning streaming on, reading the stream, ending it properly, and the case that is easy to get wrong: the client going away before the model has finished.

Turning it on

One field:

TypeScript
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.SALYRO_API_KEY,
  baseURL: 'https://api.salyro.com/v1',
});

const stream = await client.chat.completions.create({
  model: 'openai/gpt-4o-mini',
  messages: [{ role: 'user', content: 'Explain what a gateway does.' }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}

curl -N matters: without it, curl buffers the response and you watch nothing happen until the whole answer arrives, which looks exactly like streaming not working.

What comes down the wire

Chat Completions streams as server-sent events. Each event is a chat.completion.chunk carrying a delta — the piece of text produced since the last one — and the sequence ends with a literal [DONE]:

Text
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"A gateway"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" sits in front"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Three things about this shape are worth internalising:

  • A delta may be empty. The first chunk usually carries the role and no text. Concatenating without a null check produces undefined in the middle of your output.
  • finish_reason arrives on its own chunk, near the end, with no content beside it. "stop" is a complete answer; "length" means the model hit a token limit mid-thought, which is a different thing to show a user.
  • [DONE] is a literal string, not JSON. Parsing it as JSON is the classic streaming bug, and it happens on the last event of every successful stream.

Ending cleanly

A stream that ends properly has done all three of: delivered finish_reason, delivered [DONE], and closed. If you are writing the consumer yourself rather than using an SDK, treat the absence of any of them as an incomplete answer rather than as a shorter one.

The distinction matters where the text is being persisted. An answer truncated by a dropped connection and an answer that finished are indistinguishable once they are a string in a database, unless you recorded which one it was.

When the client goes away

A browser tab closes, a user navigates away, a request times out at your edge. The connection between your server and the browser drops — but the request to the provider is not automatically cancelled by that, and neither is the generation already in flight.

The practical consequence is that abandoning streams cheaply is not free. If users routinely start a generation and leave, cancel the upstream request when your own client disconnects rather than letting it run to completion:

TypeScript
// Abort the upstream request when the browser's connection to you drops.
const controller = new AbortController();
request.signal.addEventListener('abort', () => controller.abort());

const stream = await client.chat.completions.create(
  { model, messages, stream: true },
  { signal: controller.signal },
);

What is recorded

A streamed request appears in the gateway's request log like any other, with its model, provider, status and timing. What is recorded reflects what actually happened: an answer that was cut short is recorded as the tokens that were produced, not as the answer that would have been.

Records are written asynchronously, so a request that has just finished may take a moment to appear. See Logs & conversations for what a record holds, and Usage & costs for how token counts become a cost.

Streaming on the Responses endpoint

/v1/responses streams too, and it streams in its own format — typed events describing the response as it is built, rather than Chat Completions chunks.

Through an SDK the difference is mostly invisible, because the SDK owns the parsing:

TypeScript
const stream = await client.responses.create({
  model: 'openai/gpt-4o-mini',
  input: 'Explain what a gateway does.',
  stream: true,
});

for await (const event of stream) {
  if (event.type === 'response.output_text.delta') {
    process.stdout.write(event.delta);
  }
}

Remember that Responses is stateless in V1: there is no previous_response_id, no conversation object, and no way to fetch a response back after the stream ends. If you need the text, keep it as it arrives. The Public API reference sets out that boundary in full.

Streaming and other capabilities

  • Structured output works with streaming, but the JSON is only valid once it is complete. Do not parse a partial stream — buffer it, then parse. See Get structured JSON output.
  • Tool calls arrive in deltas too, assembled across chunks rather than delivered whole. Accumulate the arguments string before parsing it.
  • Failures before the stream starts are ordinary HTTP errors with a status and an error body — an unaccepted key or a rejected parameter never becomes a stream. Debug a failed request covers reading them.

Related guides