Tag: Software Development

Embeddings are how you turn text into something a computer can search by meaning rather than by matching words. An embedding model reads a string of text and returns a fixed-length vector of floats — typically a few hundred to a few thousand numbers — such that semantically similar texts produce vectors that are close together in that space. Once you have that, “find me the document most relevant to this question” becomes “find me the vector closest to this question’s vector”, and the rest of a RAG system falls into place.

This guide covers what embeddings actually are, how to call them through an AI API, how similarity search works, and how to build a minimal retrieval-augmented generation (RAG) pipeline that fetches the right context before asking a model to answer. The code runs unmodified against any OpenAI-compatible endpoint, and the patterns fit naturally into the rest of this site’s guides on cost, function calling, and rate limits.

Why embeddings matter

Keyword search finds documents that share words with the query; embeddings find documents that share meaning. The difference is felt the moment a user types something a writer never phrased that way. “How do I cancel my plan?” and “What is the cancellation policy?” have no shared words but the same answer — and an embedding search surfaces it without anyone hand-writing a synonym list.

Three properties of embeddings matter in practice:

• Similarity is geometric. Two vectors being “close” means their dot product (or cosine similarity) is high. You do not need a model at query time to find a match — a fast nearest-neighbour search over a few thousand vectors takes milliseconds.
• Embeddings are cheap to compute once you have them. A typical 1,000-token passage embeds in tens of milliseconds and costs fractions of a cent. Re-embed only when the model or the content changes.
• You embed any text the model can read. Documents, support tickets, code snippets, product descriptions — the same vector space, the same similarity function, the same retrieval.

How embeddings work in practice

When you send text to an embeddings endpoint, the provider’s model returns a JSON object with a fixed-length vector — typically 1536 or 3072 numbers for modern OpenAI models. Two texts that mean similar things produce vectors that point in similar directions; unrelated texts produce vectors that point in unrelated directions. The exact geometry is not interpretable, but the angle between two vectors reliably correlates with how semantically related the texts are.

This is the entire mental model. The arithmetic behind it (transformer encoders, contrastive training, projection layers) does not matter for using embeddings well — only the invariants do: similar in → close vectors out, different in → far vectors out.

Calling the embeddings API

The OpenAI-compatible contract for embeddings is one of the simplest in the API: POST a list of strings, get a list of vectors back. This works against OpenAI directly, against any relay that exposes the same endpoint, and against smaller specialised providers:

from openai import OpenAI

client = OpenAI()

resp = client.embeddings.create(
    model="text-embedding-3-small",
    input=[
        "How do I cancel my subscription?",
        "What is the refund policy?",
        "Today's weather in Lisbon",
    ],
)

for i, item in enumerate(resp.data):
    print(f"vector {i}: dim={len(item.embedding)} sample={item.embedding[:4]}")
# vector 0: dim=1536 sample=[0.0123, -0.0451, 0.0089, 0.0204]
# vector 1: dim=1536 sample=[0.0118, -0.0438, 0.0091, 0.0211]
# vector 2: dim=1536 sample=[-0.0302, 0.0152, 0.0612, -0.0088]

Three properties to notice. Vectors 0 and 1 (both about customer service) are very close — sample values are similar; vector 2 (about weather) is far. The model used here is a small, fast, cheap one — fine for most retrieval workloads, with a vector dimension of 1536. Larger models produce higher-dimensional vectors (3072 for the equivalent “large” variant) that can capture finer distinctions at a higher cost.

Similarity: cosine and dot product

Two ways to compare vectors: dot product (sum of element-wise products) and cosine similarity (the dot product divided by the lengths of both vectors). For normalised embeddings — those that already lie on the unit sphere — the two are equivalent. Most modern embedding models return vectors that are either pre-normalised or close enough that cosine is the right choice; computing it is straightforward:

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na  = math.sqrt(sum(x * x for x in a))
    nb  = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb)

q = client.embeddings.create(model="text-embedding-3-small",
                             input="cancel my subscription").data[0].embedding

docs = [
    "How do I cancel my plan?",
    "What is your refund policy?",
    "Today is sunny in Lisbon.",
]
doc_vecs = client.embeddings.create(model="text-embedding-3-small",
                                    input=docs).data

for text, vec in zip(docs, doc_vecs):
    print(f"{cosine(q, vec.embedding):.3f}  {text}")
# 0.873  How do I cancel my plan?
# 0.781  What is your refund policy?
# 0.412  Today is sunny in Lisbon.

The ranking is what you would expect: cancellation is closest to the query, refund policy is next (related domain), weather is far. Cosine returns a value between -1 and 1; for embeddings, anything above ~0.7 is usually a meaningful match.

Chunking: how to split documents for embedding

A 50-page PDF does not embed as one vector — embeddings models have token limits, and a single vector that tries to represent the entire document averages away the parts that matter. The standard solution is to split the document into chunks, embed each chunk separately, and retrieve the chunks that match a query. Four chunking strategies cover most production needs:

• Fixed-size windows. Split every N tokens with M tokens of overlap. Simple, predictable, and a fine default. Start with N=500, M=50.
• Sentence or paragraph boundaries. Split where the text naturally divides, keeping each chunk semantically self-contained. Better retrieval quality, less predictable size.
• Heading-based. Use the document’s own structure — markdown headers, HTML <h2> tags, PDF sections — as chunk boundaries. Excellent for structured documents.
• Semantic splitting. Embed sentences one at a time, then merge adjacent sentences whose embeddings are very similar. Sophisticated; worth it only when the simpler strategies underperform.

The overlap parameter (M in the first strategy) matters more than the size of N: it prevents a sentence that happens to span a chunk boundary from being lost. The standard pattern is small overlap (5-10%) so each chunk is mostly unique but no information is silently dropped at the seams.

Chunking strategies compared

Strategy	How it splits	Best for	Watch out for
Fixed-size	Every N tokens, with a small overlap	Homogeneous prose, quick prototypes	Cuts sentences and tables in half
Sentence / paragraph	On natural language boundaries	Articles, documentation, support tickets	Very uneven chunk sizes
Recursive	Tries paragraphs, then sentences, then characters	Most mixed-format corpora — a good default	Needs tuning of the size thresholds
Structure-aware	On headings, sections, code blocks, table rows	Markdown, HTML, and PDFs with clear structure	Requires a parser per format
Semantic	Where embedding similarity between adjacent passages drops	Dense reference material, transcripts	Extra embedding cost and complexity

A minimal RAG pipeline

RAG — retrieval-augmented generation — is just three steps on top of what you already have: embed the query, fetch the closest chunks, ask a model to respond with the chunks as context. Here is the simplest possible end-to-end version:

import numpy as np
from openai import OpenAI

client = OpenAI()

INDEX = []  # list of (chunk_text, embedding_vector) pairs

def index_documents(docs):
    """Embed and store chunks once at index time."""
    embs = client.embeddings.create(
        model="text-embedding-3-small", input=docs,
    )
    for text, item in zip(docs, embs.data):
        INDEX.append((text, np.array(item.embedding)))

def search(query, k=3):
    """Return the k chunks whose embeddings are closest to the query."""
    q = np.array(
        client.embeddings.create(
            model="text-embedding-3-small", input=query,
        ).data[0].embedding
    )
    scored = sorted(
        ((float(q @ v / (np.linalg.norm(q) * np.linalg.norm(v))), t)
         for t, v in INDEX),
        reverse=True,
    )
    return [t for _, t in scored[:k]]

def answer(question):
    context = "\n\n".join(search(question, k=4))
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content":
                "Answer using the context below. If the answer is not "
                "in the context, say you do not know."},
            {"role": "user", "content":
                f"Context:\n{context}\n\nQuestion: {question}"},
        ],
    )
    return resp.choices[0].message.content

# Index a tiny corpus once.
index_documents([
    "Refunds are issued within 7 days of cancellation.",
    "Premium plans can be cancelled any time from the dashboard.",
    "Free trials do not require cancellation to end.",
    "Weather in Lisbon today is 22 degrees and sunny.",
])

print(answer("How do refunds work?"))
# Refunds are issued within 7 days of cancellation.

Three details matter most. First, the corpus is embedded once at index time; queries only embed the user’s question. Second, the model is told to say “I do not know” when the context does not contain the answer — this is the only thing keeping RAG from hallucinating when retrieval fails. Third, the model is the same one you use for chat — the OpenAI-compatible contract covers both completions and embeddings, so a single endpoint serves both.

When you need a real vector store

The list above is fine for thousands of chunks. Past that, brute-force cosine over every vector is too slow and a real vector index pays off. The choice is between:

• Hosted vector databases. Pinecone, Weaviate Cloud, Qdrant Cloud, etc. Operate the index for you, charge by storage and queries, scale to millions of vectors without engineering work.
• Self-hosted libraries. FAISS (Meta), Annoy (Spotify), HNSW implementations in pgvector, etc. You run them yourself, often inside an existing database. Free and fast for moderate scale, but you own the operations.
• Hybrid stores. PostgreSQL with the pgvector extension, ElasticSQL with dense_vector, SQLite with sqlite-vec. Convenient when the metadata you want to filter by already lives in a relational database.

The rule of thumb: brute force is fine up to about 10,000 chunks; once you cross 100,000, a real index becomes necessary. A useful hybrid is to combine vector similarity with a metadata filter (region = “EU”, date > “2026-01-01”) to narrow the candidate set before the expensive similarity computation — most vector stores support this natively.

Common failure modes

• Mixed embedding models. If some chunks were embedded with one model and queries with another, similarity is meaningless. Pick one model and stick with it; if you migrate, re-embed the entire corpus.
• Wrong chunk size. Too small loses context (“the refund” without the qualifier); too large averages meaning across too much text. 200-500 tokens is a sensible range for prose.
• Missing overlap. A sentence that crosses a chunk boundary is split in two, and neither chunk contains its full meaning. Keep 5-10% overlap.
• Forgetting to normalise. Some vectors are returned pre-normalised, some are not. If you mix dot product and cosine without thinking, results can quietly degrade.
• Ignoring the prompt. A model given retrieved context and no instruction will confabulate when context is thin. “If the answer is not in the context, say you do not know” is the single most useful prompt line in any RAG system.

Cost and rate limits

Embeddings are cheap enough that cost is rarely the limiting factor, but the call patterns matter:

• Embed the corpus once at index time. Re-embedding on every query is wasteful. Store vectors, reuse them.
• Batch when embedding in bulk. Most embeddings APIs accept a list of inputs in one request and discount per token. A single call of 1,000 chunks is much cheaper than 1,000 calls of one chunk.
• Watch the rate limits. Indexing 100,000 chunks can hit the same RPM/TPM limits as chat completions. See our rate-limits guide for the patterns that keep large embedding jobs within quota.

Embeddings and RAG checklist

• Use a single embedding model end-to-end (corpus and queries).
• Embed chunks of 200-500 tokens with 5-10% overlap.
• Store vectors alongside a chunk identifier and a back-reference to the source.
• Cache query embeddings for repeat queries on hot paths.
• Use cosine similarity; check whether your model returns pre-normalised vectors.
• Brute-force cosine is fine up to ~10,000 chunks; switch to a vector index beyond that.
• Tell the generation model to say “I do not know” when context is thin.
• Embed in batches at index time; respect rate limits on the embedding endpoint.
• Version your index by the embedding model used so re-indexing is auditable.
• Measure retrieval quality with a small evaluation set before chasing model upgrades.

Frequently asked questions

What are embeddings in an AI API?

Embeddings are vectors — fixed-length lists of floats — produced by an embedding model from a piece of text. Semantically similar texts produce vectors that are close in the model’s vector space, so similarity becomes a fast geometric calculation. They power semantic search, retrieval-augmented generation, clustering, recommendations, and classification with very little code.

What dimension should I use?

Whatever your chosen model returns. Smaller models (1536 dimensions) are cheap, fast, and fine for most retrieval workloads. Larger models (3072 dimensions) capture finer distinctions at higher cost and storage. Pick once and stick with it; mixing dimensions across documents and queries silently breaks similarity.

Cosine similarity or dot product?

For modern embeddings models that return vectors roughly on the unit sphere, the two are equivalent in ranking. If your model does not normalise its vectors, cosine is the safer default — it normalises on the fly. For normalised vectors, dot product is faster because it skips the division.

How large should a chunk be?

For prose, 200-500 tokens is a sensible range; smaller chunks lose context, larger chunks average away meaning. Keep 5-10% overlap between adjacent chunks so sentences that straddle a boundary are not lost. The exact right size depends on your data — measure retrieval quality on a small evaluation set before optimising.

When do I need a vector database?

Brute-force cosine over a few thousand vectors is fine for prototypes and small production workloads. Past about 10,000 chunks, brute-force search starts to feel slow at query time; past 100,000, a real index is required. Hosted options (Pinecone, Weaviate Cloud) and self-hosted options (FAISS, pgvector) both work; the choice depends on whether you want to operate the infrastructure yourself.

Will a RAG system still hallucinate?

Yes, if the model is asked a question the retrieved context does not answer. The single most useful prompt line in any RAG system is telling the model to say “I do not know” when the context is thin. Strong retrieval plus that instruction dramatically reduces hallucinations; it does not eliminate them, which is why measuring retrieval quality and answer quality together is part of running RAG in production.

Are embeddings OpenAI-compatible across providers?

The /v1/embeddings endpoint is one of the most standardised parts of the OpenAI-compatible contract — most relays expose it. The catch is that vectors produced by different models are not interchangeable: even if two providers agree on the wire format, a vector from one model will not compare meaningfully to a vector from another. Pick the model and the provider, and re-embed if you ever migrate.

Are embeddings expensive?

Usually not. A typical 1,000-token passage embeds in tens of milliseconds and costs fractions of a cent on the small, fast model. The expensive part of a RAG system is almost always the generation step, not the embedding step. See our guide to reducing AI API costs for the broader patterns.

---

Embeddings turn “search” into geometry and “RAG” into three steps: embed the corpus once, embed each query at request time, ask the model to answer with the closest chunks as context. The patterns fit together because the OpenAI-compatible contract serves both completions and embeddings, so a single endpoint can power the entire stack. If you want to try embeddings and RAG against multiple models with the same code, create a key at qoraapi.com and start with a small corpus — most of what makes RAG work in production is the indexing and retrieval pipeline, not the model itself.

Related reading

• Building Production RAG: Chunking, Hybrid Search, and Re-Ranking
• How to Choose a Vector Database for RAG
• Building a RAG Ingestion Pipeline: Crawling, Parsing, and Syncing
• Semantic Caching for AI APIs: Cut Latency and Cost by Up to 60%
• Text-to-SQL: Letting Users Query Your Database with AI
• Building Voice AI Apps: TTS, STT, and Realtime APIs
• Managing the Context Window: Truncation, Summarization, and Sliding Windows

AI API streaming is what lets users see the model’s answer as it is generated rather than waiting for the whole response. The model still produces the same tokens, costs the same amount, and takes the same total time — what changes is when the bytes reach your client. For chat interfaces, autocomplete, and any interactive surface, that single change is often the difference between a product that feels alive and one that feels slow.

Under the hood, streaming is just Server-Sent Events over HTTP. The wire format is simple, every major AI provider speaks it, and the SDKs handle the parsing for you. The interesting questions are at the edges: how to handle buffered responses, how to combine streaming with function calling, what to do when a network connection drops mid-response, and how to make sure cost and reliability work stay intact when bytes arrive one chunk at a time instead of one body. This guide covers all of that, with code that runs unmodified against any OpenAI-compatible endpoint.

Why streaming matters

The total time to generate an answer is mostly model inference time, and streaming cannot shorten it — the GPU produces the same tokens at the same rate regardless. What streaming changes is time to first token (TTFT): how long the user waits before anything appears on screen. On a 500-token answer that takes four seconds end-to-end, TTFT typically drops from ~4s to ~200ms with streaming, which is the difference between “the page is broken” and “the page is working”.

Three properties of streaming matter in production:

• Same cost. You are billed for every token the model produces, delivered or not. Streaming does not reduce spend; it changes the user experience. Our guide to reducing AI API costs covers the patterns that do.
• Same total time. Inference throughput is what it is. Streaming exposes the same work over more round trips rather than running it faster.
• Faster perceived latency. This is the only thing that improves — and for chat, autocomplete, and any “the user is waiting on this” surface, it is the only thing that matters.

Streaming vs non-streaming at a glance

Dimension	Non-streaming	Streaming (SSE)
Time to first token	Waits for the entire completion	Tokens arrive as they are generated
Perceived latency	High for long answers	Low — the answer starts appearing immediately
Total cost	Same	Same
Total wall-clock time	Same	Same
Client complexity	One request, one parse	Event loop, partial parsing, reconnection logic
Best suited to	Batch jobs, classification, extraction, backend pipelines	Chat UIs, autocomplete, agents — any interactive surface
Failure handling	Retry the whole call	Must handle mid-stream drops and resume cleanly
Load-balancer impact	Short-lived connections	Long-lived connections — check idle timeouts

How Server-Sent Events work on the wire

An AI API streaming response is an HTTP response with Content-Type: text/event-stream. The body is a stream of events, each event formatted as one or more field: value lines terminated by a blank line. The server flushes events as soon as they are ready, and the client parses them as they arrive.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hel"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"lo"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":","},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" world"},"index":0}]}

data: [DONE]

Each data: line carries a JSON payload describing one chunk of the answer — typically a small delta containing the new tokens since the last chunk. The final event is data: [DONE], which is the signal that the response is complete. There is no event: field for chat completions; some other endpoints (like Assistants) use named event types for state transitions.

Consuming streams with the official SDKs

The OpenAI Python and Node SDKs handle SSE for you. You set stream=True on the request and iterate over the response — each iteration gives you a parsed chunk, no manual framing required:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a one-line haiku about streaming."}],
    stream=True,
)

full = []
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        full.append(delta)
        print(delta, end="", flush=True)
print()                              # newline after the stream completes
print("complete:", "".join(full))

The same call against an OpenAI-compatible endpoint works unchanged — just point the client at the relay’s base_url. Under the hood the SDK is doing exactly what you would do by hand: open an HTTP request, read the response line by line, parse each data: line as JSON, and yield the parsed object.

Consuming streams with raw fetch

Sometimes you do not want to pull in an SDK — for a serverless function, a small worker, or a custom client. Plain fetch with a streaming body works just as well. The pattern below runs in any modern JavaScript runtime and is what the SDK is doing internally:

const resp = await fetch("https://qoraapi.com/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${apiKey}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Stream a one-line haiku." }],
    stream: true,
  }),
});

const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });

  // SSE events are separated by a blank line. Split on "\n\n"
  // and process every complete event in the buffer.
  let boundary;
  while ((boundary = buffer.indexOf("\n\n")) !== -1) {
    const event = buffer.slice(0, boundary);
    buffer = buffer.slice(boundary + 2);

    for (const line of event.split("\n")) {
      if (!line.startsWith("data:")) continue;
      const payload = line.slice(5).trim();
      if (payload === "[DONE]") return;
      const json = JSON.parse(payload);
      const delta = json.choices?.[0]?.delta?.content ?? "";
      process.stdout.write(delta);
    }
  }
}

The buffer is the key detail: SSE events are framed by blank lines, but a single network read can deliver a partial event, multiple events, or a multi-byte UTF-8 character split across two reads. Code that splits on "\n\n" without a buffer will silently truncate or corrupt the last chunk of every response.

Streaming with function calling

Streaming and function calling compose, but the way they compose matters. The model’s tool call is delivered as a complete structured object, not as a token stream — what streams is the reasoning before the call and the final text after the result. The OpenAI SDK accumulates the tool call into a complete object once the stream ends:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools,
    stream=True,
)

# Accumulate the deltas into a single tool call.
tool_call_chunks = []
for chunk in stream:
    for delta in chunk.choices[0].delta.tool_calls or []:
        tool_call_chunks.append(delta)

# Stitch the streamed chunks into a complete tool_call.
final = tool_call_chunks[0]                          # shape stays the same
final.function.arguments = "".join(
    c.function.arguments for c in tool_call_chunks
)

If you want streaming for the surrounding text, this is the right shape: stream the visible content into the UI, accumulate the tool call in the background, and dispatch the function once the stream ends.

Common pitfalls in production

Most streaming bugs are not about the SDK. They come from the layer between the provider and your client — and they are worth knowing about before they reach production:

• Proxy buffering. Many HTTP proxies (Cloudflare, nginx with proxy_buffering on, certain load balancers) buffer the entire response before passing it on. This silently turns streaming into a non-streaming response, and users see a long pause followed by the full answer. The fix is at the proxy: set X-Accel-Buffering: no for the streaming path, or use HTTP/1.1 with Transfer-Encoding: chunked.
• Gzip compression. A gzipped SSE response can look like garbage on a partial read because the gzip header is at the start and the body grows as new chunks arrive. Make sure your client supports streaming decompression, not a one-shot inflate.
• Lost connections. Mobile networks and aggressive proxies close idle HTTP connections. A long generation can exceed the idle timeout, and the client never sees [DONE]. Either send keep-alive pings, set a server-side timeout that flushes a chunk periodically, or accept that very long streams need reconnect logic.
• Truncated JSON. Each data: line must be parsed independently. If a read delivers half a JSON payload, do not try to parse it — wait for the next chunk. Aggregating partial JSON across reads is one of the easiest ways to introduce subtle corruption.
• Backpressure. If your consumer is slower than the producer, events buffer up in memory and eventually OOM. Real systems either enforce a max-buffer size, drop old events, or pause the producer.

Reliability, retries, and rate limits

Streaming responses do not play nicely with retries: a connection that drops mid-response has already produced some tokens, and the user has already seen them. Retry policies that assume the request either completed or did not begin will over-charge on streaming. Three rules keep this honest:

• Retry on connection errors before any bytes arrive. Once you have displayed a single token to the user, retrying the same request will bill the model twice for the same answer.
• Do not retry once the stream ends. If the stream completed (you saw [DONE] or a finish reason), the request succeeded; a retry is a duplicate.
• Back off with jitter when the provider throttles. A 429 with retry-after still applies to streaming. See handling AI API rate limits for the broader pattern.

Streaming checklist

• Enable stream: true on every interactive surface; leave it off for background jobs.
• Render tokens to the UI as they arrive, not in one batch at [DONE].
• Buffer SSE frames across reads — never assume a network read delivers a complete event.
• Configure the proxy (Cloudflare, nginx, load balancer) to flush streaming responses immediately.
• Support streaming decompression if the response is gzipped.
• Retry only before the first byte has been emitted to the user; never after.
• Set a max_tokens ceiling so a runaway stream cannot bill indefinitely.
• Surface a clear error when the connection drops before [DONE] instead of leaving the UI in limbo.
• If you combine streaming with function calling, accumulate the tool call across the stream and dispatch once the model finishes.
• Watch rate limits and back off with jitter — streaming responses still count against your quotas.

Frequently asked questions

Does streaming reduce AI API costs?

No. You are billed for every token the model produces regardless of how the response is delivered. Streaming changes when the bytes reach you, not how many tokens the model emits. See our guide to reducing AI API costs for the patterns that do.

Is streaming faster end-to-end?

No. The total time to generate a response is the same — streaming only shortens the wait before the first token arrives. For a 500-token answer that takes four seconds, total time is still four seconds, but the user sees the first token in roughly 200ms instead of after the full generation.

Do I need an SDK to consume a stream?

No. Plain fetch with a streaming body works fine — every modern runtime exposes a ReadableStream or equivalent. The SDKs add conveniences like automatic buffering and tool-call accumulation, but the wire protocol is just Server-Sent Events and can be parsed with a few lines of code.

Why does my streaming response arrive all at once?

Almost always a buffering proxy. Cloudflare, nginx (with proxy_buffering on), and most load balancers buffer the entire response before forwarding it, which silently turns streaming into non-streaming. The fix is at the proxy: X-Accel-Buffering: no for nginx, or a streaming-friendly Cloudflare configuration for that path.

Can I stream with function calling?

Yes — but the tool call itself is delivered as a complete structured object, not as a stream of tokens. What streams is the model’s reasoning before the call and the final text after. Accumulate the tool-call deltas across the stream and dispatch the function once the stream ends. See our function-calling guide for the full pattern.

Can I retry a streaming request that drops mid-response?

Only if no bytes have reached the user yet. Once the first token is on screen, retrying the same request bills twice for the same answer. The pattern is: retry on connection error before any UI rendering, and surface a “connection lost” message after.

How do I know how many tokens a stream produced?

Most providers include a final chunk with usage statistics (prompt, completion, total tokens). The OpenAI SDK returns it on the last chunk under chunk.usage. Capture it there, log it, and use it for cost tracking — see our cost guide for how to wire that into a per-task dashboard.