The context window is a budget, not a bucket: input and output tokens draw on the same limit, and cost scales with everything you send. Manage it with four levers — truncation, summarization, retrieval-on-demand, and routing — plus a per-section token budget and an offline eval loop that tells you when more context stops helping.
The techniques below are ordered by marginal cost, because the cheapest fix is almost never “call a model to fix it.”
Why the context window is a budget, not a bucket
A model with an N-token window does not give you N tokens of input. It gives you N tokens of input plus output. If you send M input tokens, the maximum completion you can request is N − M. Teams discover this the hard way: they set max_tokens=4000, send a 6,000-token prompt to an 8,192-token model, and get a 400 back. Worse, some SDKs and wrappers silently trim the prompt instead of failing, so the model answers confidently with the last third of your document missing.
The correct sequence is reserve, then fill: subtract the output reserve from the window first, and treat the remainder as the input budget. Never assemble the prompt and hope it fits.
The second reason it is a budget is that cost scales with input, and input compounds. A chat request re-sends the entire prefix on every turn. Turn 20 re-bills turns 1 through 19. Pricing is linear in tokens, but the tokens per turn grow with conversation length, so the cost of a session grows roughly quadratically with the number of turns. That is the real reason long conversations get expensive — not the output.
- Unbounded history: session cost is O(turns²) in input tokens, and time-to-first-token (TTFT) grows every turn as prefill lengthens.
- Bounded history of K turns: session cost becomes O(turns × K) — linear again, with flat TTFT. This single change is usually the largest cost win available.
- Attention compute grows super-linearly with sequence length, so a 4× longer prompt costs more than 4× to process even at a flat per-token price.
So “the window is 200K, I can send 200K” is a budgeting error, not a feature — your effective window excludes the output reserve, and recall degrades before you reach the hard limit.
Strategies when you exceed the budget
There are five levers, and applying them in the wrong order is why teams end up paying a summarization call on every turn. Escalate in increasing marginal cost: deterministic transforms first, model calls last, infrastructure changes only when the data justifies it.
| Lever | What it does | Marginal cost | Use when | Failure mode |
|---|---|---|---|---|
| Drop oldest (turn-boundary truncation) | Removes the oldest messages, keeping the newest K turns verbatim | Zero — no extra call, no latency | Chat where early turns are genuinely stale; the default first move | Silently deletes a constraint the user set at turn 2, so the model contradicts it later. Mitigate with a pinned-facts list. |
| Rolling summarization | An LLM compresses dropped turns into a bounded summary carried forward | One extra call, ~5–10% of the summarized tokens, plus latency | Long sessions where early decisions still matter | Recursive drift: summarize a summary enough times and details mutate, then get hallucinated. |
| Retrieve on demand (agentic memory) | Nothing is pre-stuffed; the model calls a search tool over full history or a document store | Only when invoked — a tool round-trip on those turns | Histories or corpora that can never fit whole | The model cannot know what it does not have. Without an index or hint in the prompt, it never thinks to look. |
| Compress in place | Deterministic pruning: strip HTML boilerplate, collapse whitespace, dedupe repeated chunks, normalize JSON | Near zero — runs in your process | Always, before anything else. Frequently 20–60% off the prompt for free | Irreversible. Never compress instructions, constraints, or schema definitions — only bulk content. |
| Route to a bigger window | Same prompt, a model with a larger context limit | Per-token price usually rises; input cost scales with the bigger prompt | Genuine long-document tasks that must be read whole | A bigger window is not better recall. Cost rises immediately, quality often does not. |
The decision rule: compress deterministically, then drop oldest, then summarize, then route. Escalate only when eval data shows the current lever costs you accuracy. Retrieval-on-demand is less a later lever than a different architecture — if your corpus is a knowledge base rather than a conversation, it is the right first answer, and it is the pattern behind embeddings and RAG.
One constraint cuts across all five: never split a tool call from its result. Truncating a message whose tool_call was dropped produces orphaned tool messages, and most providers reject the request outright. Every cut point must land on a safe boundary — the code below does this.
Sliding window plus rolling summary memory
The pattern that survives production combines a sliding verbatim window (recency is what users actually reference), a rolling summary for everything older, and a pinned-facts list that is never summarized away. Pinning is the fix for drift — names, IDs, units, and explicit constraints bypass summarization entirely.
import tiktoken
ENC = tiktoken.get_encoding("o200k_base") # match your target model's tokenizer
def count_tokens(text: str) -> int:
return len(ENC.encode(text))
SUMMARY_PROMPT = """You maintain the running memory of a long conversation.
Merge EXISTING SUMMARY with NEW TURNS into one summary.
KEEP: decisions, user-stated facts, constraints, open questions, IDs, names, units.
DROP: pleasantries, restated context, anything already obvious.
Output terse bullets only, at most {limit} tokens."""
class BoundedHistory:
"""Sliding verbatim window + rolling summary + pinned facts + hard token cap."""
def __init__(self, system, client, summarize_model="gpt-4o-mini",
window_turns=8, summarize_at=16, summary_budget=400,
max_input_tokens=12_000):
self.system = system
self.client = client
self.summarize_model = summarize_model
self.window_turns = window_turns # turns kept verbatim
self.summarize_at = summarize_at # fold once history exceeds this
self.summary_budget = summary_budget
self.max_input_tokens = max_input_tokens
self.summary = ""
self.pinned = [] # never summarized away
self.turns = []
def pin(self, fact: str) -> None:
"""Call whenever the user states a durable constraint or identifier."""
self.pinned.append(fact)
def _summarize(self, dropped) -> None:
convo = "\n".join(f"{m['role']}: {m['content']}" for m in dropped)
r = self.client.chat.completions.create(
model=self.summarize_model,
messages=[
{"role": "system",
"content": SUMMARY_PROMPT.format(limit=self.summary_budget)},
{"role": "user",
"content": f"EXISTING SUMMARY:\n{self.summary or '(none)'}\n\n"
f"NEW TURNS:\n{convo}"},
],
max_tokens=self.summary_budget,
)
self.summary = r.choices[0].message.content.strip()
def _safe_cut(self, cut: int) -> int:
"""Move the cut back so a tool call is never separated from its result."""
while cut > 0 and self.turns[cut - 1].get("role") == "tool":
cut -= 1
return cut
def add(self, role: str, content: str) -> None:
self.turns.append({"role": role, "content": content})
# Fold oldest turns into the summary in batches, amortizing the extra call.
while len(self.turns) > self.summarize_at:
cut = self._safe_cut(len(self.turns) - self.window_turns)
if cut <= 0:
break
self._summarize(self.turns[:cut])
self.turns = self.turns[cut:]
def _head(self):
"""System, summary, and pins sit at the TOP of the prompt, not the middle."""
head = [{"role": "system", "content": self.system}]
if self.summary:
head.append({"role": "system",
"content": "Conversation summary so far:\n" + self.summary})
if self.pinned:
head.append({"role": "system",
"content": "Pinned facts (authoritative):\n- "
+ "\n- ".join(self.pinned)})
return head
@staticmethod
def _count(msgs) -> int:
return sum(count_tokens(m["content"]) for m in msgs)
def build(self, user_msg: str, output_reserve: int = 800):
head = self._head()
budget = self.max_input_tokens - output_reserve # reserve first
turns = list(self.turns)
msgs = head + turns + [{"role": "user", "content": user_msg}]
while turns and self._count(msgs) > budget: # clamp if still over
turns = turns[1:]
msgs = head + turns + [{"role": "user", "content": user_msg}]
used = self._count(msgs)
if used > budget:
raise ValueError(f"context budget exceeded: {used} > {budget}")
return msgs, output_reserve
# Usage
mem = BoundedHistory(system="You are a support engineer for the billing API.",
client=client)
mem.pin("Customer is on the Enterprise plan, billed annually.")
mem.add("user", "Our invoice shows a duplicate charge for March.")
mem.add("assistant", "I can see two line items. Let me pull the ledger.")
messages, reserve = mem.build("What should I tell finance?")
resp = client.chat.completions.create(model="gpt-4o", messages=messages,
max_tokens=reserve)
Four parameters carry the whole design. Tuning notes that matter in practice:
summarize_atshould be roughly 2×window_turns. Folding on every turn pays a summarization call per message and thrashes the summary. Batching at double the window amortizes it, at the cost of a temporary prompt spike.- Summarize with a small model. Compression is not reasoning, and the summary budget caps output anyway.
- Chain as prior summary + newly dropped turns, never summary-of-summary alone. Each pass must see raw text for the new material, or drift compounds fast.
- Pin aggressively, summarize reluctantly. Drift comes almost entirely from facts that were summarized twice.
- Emit metrics per build:
prompt_tokens,summary_tokens,dropped_turns, and whether the clamp fired. A clamp that fires on every request means your budget is wrong, not that your code is safe.
For multimodal content, count_tokens needs a branch: image and audio parts are billed in units that are not characters, and the provider’s usage report is the only reliable count.
Token budgeting per section
A budget you can defend has a number for every section and headroom you did not spend. Here is a working budget for a support assistant on a 128K-window model, deliberately capped at 16,000 input tokens because the eval data showed no accuracy gain above it:
- System prompt + policies: 900
- Tool schemas (5 tools): 700
- Pinned facts: 300
- Rolling summary: 600
- Retrieved knowledge chunks (8 × ~700): 5,600
- Verbatim history (last 12 turns): 3,000
- Current user turn + attachment: 1,500
- Subtotal — input: 12,600
- Output reserve (
max_tokens): 2,000 - Total against the 16,000 cap: 14,600 → 1,400 headroom
Tool schemas are the silent eater. Five tools cost 700 tokens; a full MCP catalog can cost 5,000–10,000 before a single user word is sent, and it is re-billed every call. Prune the tool list per request to what the turn plausibly needs — see AI function calling and tool use for the selection pattern.
The cap sits below the model’s limit, on purpose. The 1,400-token headroom absorbs a tool result you did not plan for, a retry with an error appended, or a user paste. Budget to the hard limit and every surprise becomes a failed request.
Count with the target model’s tokenizer, and re-tune when you switch models. Characters-divided-by-four is fine for English prose and wrong for JSON, code, and CJK text, where a character can cost a full token or more. If you route across providers, budget in the units of the largest tokenizer in the pool. Enforce it with an assertion in the prompt assembler, not with discipline: assert prompt_tokens <= SECTION_BUDGET fails in CI, while a review comment fails in production.
The “lost in the middle” effect
Long-context models do not attend uniformly. Retrieval accuracy across a long prompt follows a U-shaped curve: content at the beginning and the end is recalled far more reliably than content buried in the middle. This was measured systematically in 2023 and remains visible in current long-context models — it is a property of how attention distributes, not a bug a bigger window fixes.
- Constraints first. System prompt, hard rules, and output format go at the top. Never bury a constraint in the middle of a 2,000-token system prompt; split it into a short always-on core plus policies loaded on demand.
- Restate the task at the end. Put the user’s question after the retrieved context, not before it. Repeating it once more immediately before generation is the highest-return change available — ~20 tokens, and the instruction lands in the high-recall zone.
- Bookend your retrieved chunks. Sort by relevance, place the top chunks first and last, fill the middle with lower-ranked material. Or cap at four to six chunks: twenty mediocre chunks dilute two good ones.
- Few-shot examples are the most vulnerable. Examples placed mid-prompt get ignored — move the most representative one to the end.
Measuring quality vs context size
More context is not monotonically better. Accuracy is typically concave in context size: it rises as you add the evidence the task needs, plateaus, then declines as irrelevant material competes for attention. A half-day of measurement tells you where the peak is.
- Build a labeled set from real traffic. 50–100 logged tasks with the expected answer and a note on the minimum evidence required. Synthetic sets miss the messy inputs that actually break you.
- Plant a needle at depth. Include a fact that must be recovered, placed at roughly 10%, 50%, and 90% of the assembled prompt. This doubles as your lost-in-the-middle regression test.
- Run a context sweep. Same model, same temperature, same tasks at 2K / 8K / 32K / full. Only context size changes.
- Score five numbers, not one: task accuracy, contradiction rate against pinned facts, p50/p95 latency, mean input tokens, and cost per resolved task.
- Find the knee. Plot accuracy and cost-per-resolved-task against context size. Most apps peak well below the model’s maximum, often by an order of magnitude.
- Ablate one section at a time. Remove retrieved context, then the summary, then the history. If accuracy does not move when a section disappears, that section is pure cost — delete it.
- Gate it in CI. The prompt assembler is a pure function: assert the token count is within budget, the section order is stable, and no tool call is orphaned.
The metric to optimize is quality per 1,000 input tokens, not raw accuracy. A configuration scoring 2% lower on accuracy at a third of the input cost is usually the better product decision, and it compounds — smaller prompts also mean lower latency, which users notice. This is the same methodology you use to reduce AI API costs without degrading output.
How a gateway helps
Context management is a routing problem with one extra input: prompt size. Measure the assembled prompt before sending it, and token count becomes a routing key alongside task type.
- Context-aware routing. Under the threshold, send to the cheap mid-tier model. Over it, or for whole-document reads, send to a long-context model. This is the same table as model routing, with
prompt_tokensas the deciding column. - One key, many windows. Without a gateway, every long-context model is a separate SDK, base URL, auth scheme, and error shape — so “route to a bigger window” becomes an integration project instead of a string change.
- Normalized overflow errors. Providers signal context-length failures inconsistently. One predictable overflow error makes your handler a single branch that runs the compression ladder and retries, instead of a provider-specific switch statement.
- Consistent usage reporting. Uniform
prompt_tokens/completion_tokensacross models gives you the per-call budget actuals your eval loop needs. - Fallback that respects the budget. When the preferred model is throttled, the fallback must also fit the prompt. Exposing each model’s window lets you filter the fallback chain by capacity instead of discovering the mismatch as a 400.
That is the case for an AI API relay: one OpenAI-compatible endpoint in front of models with different windows and prices, so context-aware routing becomes a configuration change rather than a rewrite. qoraapi.com exposes many models through a single key, which is what makes the routing table above deployable in an afternoon.
Frequently asked questions
What actually happens when a request exceeds the context window?
Most providers return a 400-class error naming the context length, and no output is billed. The dangerous case is a client library or proxy that trims the prompt to fit instead of failing — you get a confident answer computed from a silently truncated document. Never rely on the SDK to enforce your budget: count tokens before the call, reserve the output allowance, and fail closed in your own code.
Should I just use a long-context model and skip summarization?
Only for tasks that genuinely require reading a document whole. Input cost scales with everything you send, prefill latency grows with it, and recall degrades in the middle of long prompts regardless of the advertised window. Treat long context as a deliberate route for specific tasks, not a substitute for a memory strategy — and validate it against a smaller-context configuration on cost per resolved task before committing.
How often should I summarize the conversation history?
Fold older turns when the verbatim history exceeds roughly twice the number of turns you keep in the window. Summarizing every turn pays a model call per message and accelerates drift; summarizing too rarely lets the prompt spike before it folds. The 2× ratio amortizes the call while bounding the spike.
Can truncating history break tool calling?
Yes, and it is the most common cause of mysterious 400s in agent loops. If you drop an assistant message containing a tool_call but keep the matching tool result, the message list is malformed and most providers reject it. Adjust your cut point backwards to a safe boundary so each tool call and its result stay together, and treat the pair as one indivisible unit when counting turns.
Conclusion
Treat the context window as a budget you allocate, not a bucket you fill. Reserve the output allowance first, cap history at a fixed number of turns so session cost stays linear, keep a sliding verbatim window with a bounded rolling summary and a pinned-facts list, and put instructions at the start and the end rather than the middle. Then measure: sweep context size against accuracy and cost per resolved task, ablate each section, and keep only what earns its tokens.
Ready to wire it up? Start with the AI API gateway guide and the OpenAI-compatible API explainer, then drop the BoundedHistory class above into your call path.
Leave a Reply