Prompt Caching Explained: How to Cut Costs on Repeated Context

Prompt caching stores the model’s attention key/value (KV) state for a stable prompt prefix, so a repeated system prompt or long document is processed and billed once instead of on every call. You get lower time-to-first-token and a smaller input bill — but only if the prefix never changes.

The catch is that “never changes” is stricter than most codebases assume. This guide covers what providers actually cache, how to structure a prompt for guaranteed hits, TTL refresh behavior, how to read savings out of usage fields, and the failure modes that silently turn a cached prefix back into full-price prefill.

What prompt caching actually caches

Every transformer request runs in two phases. Prefill reads the entire prompt and computes attention, building a KV tensor for every token. Decode then emits output tokens one at a time, reusing those tensors. Prefill cost grows roughly quadratically with prompt length, which is why a 30k-token instruction block adds seconds before the first token appears — and why it dominates the input bill on chatty workloads with short outputs.

Prompt caching keeps the KV tensors produced during prefill for a prefix and lets a later request resume from them. On a hit, the provider skips prefill for the cached span and only prefills the uncached suffix. Three properties follow, and they explain almost every surprise you will hit later:

• It is prefix-exact, not semantic. Matching is token-level and anchored at position 0. Change token 12 of your system prompt and everything after it is cold. There is no embedding, no similarity threshold, no fuzziness — and that is the whole point.
• It caches computation, not answers. Output is still sampled fresh on every call. A cached prefix does not make responses deterministic, and it is not a substitute for an application-level response cache.
• It is monotonic. The longest matching prefix wins. If 80% of your prefix matches, you are billed and prefilled only for the 20% that missed — so a prefix that drifts by one field degrades to near-zero savings rather than partial ones.

Providers expose this through two different surfaces. Some cache automatically: any request whose prompt clears a minimum length gets its longest matching prefix cached with no markup at all. Others require explicit cache breakpoints — inline markers that declare where a cacheable span ends. Breakpoints give you control and are the safer design target, because automatic caching is a bonus, not a contract. Providers can change its minimum length, granularity, or eviction policy without notice.

Prompt caching vs semantic caching: two different layers

These two terms get conflated constantly, and conflating them leads to the wrong fix. Semantic caching is an application-layer response cache: you embed the incoming query, search a vector store for a near-duplicate, and return the stored answer without calling the model at all. Prompt caching is a provider-layer compute cache: the model still runs, but it skips re-reading a prefix it has already seen.

Dimension	Prompt caching	Semantic caching
What is cached	KV attention tensors for a token prefix	The final response text for a query
Hit condition	Byte-identical prefix from position 0	Embedding similarity above a threshold
Who controls the key	The provider — you only control prefix stability	You — threshold, normalization, TTL, invalidation
Where it lives	Provider infrastructure	Your infrastructure (vector store + app code)
Does the model run?	Yes — decode always runs	No — the call is skipped entirely
Latency win	Removes prefill, so it lands in time-to-first-token	Removes the whole round trip
Best-fit workload	Long stable instructions, tools, or corpora	Repeated user questions in varied wording
Failure mode	Any volatile byte in the prefix	Wrong answers served from a bad threshold

The practical consequence is that they solve different problems and compose cleanly. Semantic caching eliminates calls; prompt caching makes the calls you still have to make cheaper and faster. If your hit-rate problem is “users ask the same thing in different words”, you want the application-layer approach — our guide to semantic caching covers thresholds and invalidation. If your problem is “every call carries the same 20k-token instruction block”, no similarity threshold will help you: the queries are all different, and the shared part is the prefix.

How to structure prompts for cache hits

One rule generates the entire layout: stable bytes first, variable bytes last. Order every request as [stable instructions + tool schemas + fixed corpus] → [few-shot examples] → [variable user turn]. A cached span must start at position 0 and extend to a breakpoint, so you can never cache a block in the middle while leaving an earlier block volatile.

# Cache-friendly request layout: one stable prefix, one variable suffix.
#
#   |<--------------- cached prefix --------------->| variable |
#    system rules | tool schemas | fixed corpus | examples | user turn
#                              ^breakpoint      ^breakpoint

SYSTEM = render("prompts/triage_system.j2")      # ~800 tokens, changes on deploy
TOOLS  = sorted_tool_schemas()                   # ~1,500 tokens, frozen order
CORPUS = read_policy_corpus()                    # ~9,000 tokens, changes weekly

def build_messages(user_turn: str, retrieved: list[str]):
    prefix = SYSTEM + "\n\n" + render_tools(TOOLS) + "\n\n" + CORPUS
    return [
        {"role": "system", "content": [
            {"type": "text", "text": prefix,
             "cache_control": {"type": "ephemeral"}},      # breakpoint: cache ends here
        ]},
        # Per-query retrieval sits AFTER the stable block, never above it.
        {"role": "user", "content": "\n".join(retrieved) + "\n\n" + user_turn},
    ]

The rules that keep that prefix stable are mechanical, and each one maps to a real miss:

• Never interpolate volatile values into the prefix. “Current date: 2026-09-17”, request IDs, session IDs, tenant names, and experiment bucket labels all produce a unique prefix per request. Move them into the user turn.
• Serialize tool schemas deterministically. Tool definitions are part of the prefix. If your registry builds its dict from a set, or two services order the same tools differently, the token stream differs and the cache misses. Sort by name and freeze the list at startup.
• Put retrieved context after the stable block. RAG chunks change on every call; if they sit above your instructions, nothing above them can ever be cached.
• Keep template output byte-identical. A macro that emits a variable number of blank lines, or an f-string that renders None on one path and "" on another, changes the prefix even though the prompt “looks” the same in a diff.
• Respect the minimum cacheable length. Providers commonly require on the order of 1,024 tokens before anything is cached, with granularity in blocks of roughly 128 tokens. A 400-token system prompt caches nothing, however you structure it.
• Prefer fewer, larger breakpoints. One breakpoint at the end of a long stable block costs a single write and covers everything before it. Sprinkling breakpoints through a prompt multiplies writes without adding hits.

The decision criterion for whether caching is worth the work is a simple inequality: the prefix must be reused enough times inside the TTL window to amortize the cache-write surcharge. A write typically carries a modest premium over normal input (on the order of +25%), while a read is billed at a small fraction of it (on the order of a tenth). A prefix read ten times has paid for itself many times over; a prefix read once is pure overhead. As a working heuristic: if the stable prefix is over a couple of thousand tokens and reused several times within a few minutes, cache it. If it is short or used once an hour, do not.

Cache TTL and refresh behavior

A cache is only useful while it is warm, so TTL behavior shapes your architecture as much as prompt structure does. Providers cluster into three families:

Family	How you enable it	Typical lifetime	Refresh on hit
Automatic prefix caching	Nothing — a matching prefix above the minimum	Minutes of inactivity	Yes — each hit extends the window
Explicit breakpoints	Inline markers in the request	Short default (minutes); extended option around an hour at higher write cost	Yes
Explicit cache objects	Create a cache resource, reference it by ID	You set the TTL; storage is billed for its lifetime	No — you renew it yourself

Two consequences matter. First, refresh-on-hit means a steady request stream keeps a prefix warm indefinitely: at even a few requests per minute the TTL never expires, and you never pay the write surcharge again after the first one. Second, bursty traffic behaves completely differently. If a service handles a burst at 09:00 and then nothing until 11:00, the prefix expires in between and every burst opens with a cold write. For that shape, either extend the TTL deliberately or fire a scheduled no-op request every few minutes to keep the prefix alive — a keep-alive call is cheaper than a cold write on the critical path.

Scope is the other half of the story. Caches are keyed per provider, per model, and typically per organization or API key, and they are not shared across those boundaries. Two services calling the same model with different keys maintain two independent caches and pay two write costs for the identical prefix. Changing the model string starts cold as well, which is why a model canary can look like a caching regression when it is really just a cold window.

Measuring hit rate and savings

Never infer cache performance from latency alone. Every provider reports the truth in the response usage object, though the field names differ — cached input tokens, cache-creation tokens, and total input tokens. Normalize them once at the edge of your client and log the result with every call:

def cache_stats(usage) -> dict:
    """Normalize cache usage across provider response shapes."""
    details = getattr(usage, "prompt_tokens_details", None)
    read = getattr(details, "cached_tokens", 0) if details else 0
    read += getattr(usage, "cache_read_input_tokens", 0) or 0
    written = getattr(usage, "cache_creation_input_tokens", 0) or 0

    total = usage.prompt_tokens
    uncached = max(total - read, 0)
    return {
        "input_tokens": total,
        "cache_read": read,
        "cache_write": written,
        "uncached": uncached,
        "hit_rate": read / total if total else 0.0,
    }

# Cost in units of normal input price, using published relative ratios.
WRITE_PREMIUM = 1.25   # cache write vs. normal input
READ_DISCOUNT = 0.10   # cache read vs. normal input

def effective_input_units(s):
    return s["uncached"] + s["cache_write"] * WRITE_PREMIUM + s["cache_read"] * READ_DISCOUNT

def savings_vs_baseline(s):
    baseline = s["input_tokens"]
    return 1 - effective_input_units(s) / baseline if baseline else 0.0

Dashboard three series per prompt template, never globally: hit rate, cache-read tokens per request, and p95 time-to-first-token split by cached versus uncached requests. A single aggregate hit rate hides the one template that misses 100% of the time because of a date stamp. Also track the write-to-read ratio. In steady state you should see many reads per write; if writes roughly equal reads, the prefix is either expiring between requests or changing shape, and you are paying the write surcharge without amortizing it. That ratio is the earliest warning that something upstream is mutating your prefix.

Expect the latency win to land almost entirely in time-to-first-token rather than total duration. Prefill is what caching removes, and prefill happens before the first token. A workload with a long prompt and a short completion can see TTFT fall by more than half; a workload with a 200-token prompt sees nothing, because there is nothing to skip. If your completion is long, total duration barely moves even when the cache is working perfectly — do not read that as a failure.

Gotchas that silently kill the cache

Each of these presents as “caching just does not work here”, and each has a specific cause you can confirm from usage data:

• Volatile content in the prefix. Symptom: hit rate near zero from day one. Cause: a timestamp, “today is…”, or a build banner at the top of the system prompt. Fix: move every volatile string into the user turn.
• Per-request metadata in the prefix. Symptom: hit rate falls as traffic diversity rises. Cause: user ID, tenant, locale, or A/B bucket interpolated into the system prompt. Fix: send it as a suffix line or a request header.
• Reordered tools or schema keys. Symptom: intermittent misses that correlate with deploys or process restarts. Cause: dict iteration order coming from a set, or JSON serialized without a stable key order. Fix: sort deterministically and assert the rendered prefix hash in a test.
• Retrieved context above the instructions. Symptom: caching works in staging, never in production. Cause: RAG chunks injected at the top of the prompt. Fix: instructions first, retrieval last, immediately before the user turn.
• A prefix below the provider minimum. Symptom: usage reports zero cached tokens despite a stable prefix. Cause: the prompt is shorter than the minimum cacheable length. Fix: check usage before debugging anything else — a short prompt is not a bug.
• Model or version churn. Symptom: savings appear after a deploy, then vanish. Cause: the cache is keyed to the model, so a canary or version bump starts cold. Fix: roll model versions deliberately and budget for a cold window.
• Low reuse. Symptom: high cache-write tokens, low cache-read tokens. Cause: the prefix is reused less often than the TTL. Fix: shorten the TTL, batch the workload, or stop caching that prefix entirely.

Combining prompt caching with a gateway

Most of the gotchas above are consistency failures, and consistency is exactly what a gateway is good at. If five services each assemble their own system prompt and call a provider with their own key, you get five slightly different prefixes and five independent caches — five write surcharges for work that should have been done once. Put the prompt template and the API key behind one endpoint and you get one prefix, one warm cache, and one place to lint.

• One key, one cache scope. Cache scope is per credential on most providers, so centralizing the key makes every caller share the same warm prefix instead of funding their own.
• A versioned template registry. Render prompts from a single shared template, so a byte-identical prefix is a property of the system rather than something you hope each service reproduces.
• Prefix fingerprinting in CI. Hash the rendered prefix and fail the build when it changes unexpectedly — the fastest way to catch a “harmless” template edit that would have cost you the cache.
• Normalized usage metrics. A gateway sees every response and can map provider-specific usage fields into the hit-rate and write-to-read metrics above, without changing each service.
• Failover reality check. Caches do not travel between providers. Failing over to a second provider starts cold — a real cost of resilience. Keep the fallback’s prompt shape identical so its prefix is reusable once warm.

# CI guard: fail the build if the stable prefix silently changes.
import hashlib, json

PREFIX_HASH = "3f9c1a7d2b40"  # committed next to the template

def prefix_fingerprint(system: str, tools: list, corpus: str) -> str:
    blob = json.dumps([system, tools, corpus], sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()[:12]

def assert_prefix_stable(system, tools, corpus):
    got = prefix_fingerprint(system, tools, corpus)
    assert got == PREFIX_HASH, (
        f"cache prefix changed: {got} != {PREFIX_HASH}. "
        "Update PREFIX_HASH only after verifying the cache still hits."
    )

That is the practical case for routing through a single OpenAI-compatible endpoint: one key, one template, one set of metrics across every model you use. qoraapi.com exposes multiple providers behind one key, which is the cheapest way to keep one cache-friendly prefix warm across a model fleet — and it makes the usage normalization above a solved problem instead of a per-provider chore. Combine it with the other levers in our guide to reduce AI API costs; caching, routing, and token budgeting compound rather than compete.

Frequently asked questions

Does prompt caching change the model’s output?

No. Caching skips recomputing attention over a prefix that is already known; decoding is unchanged. Sampling still happens per request, so two calls sharing a cached prefix can return different completions at non-zero temperature. If you need identical responses, that is an application-level response cache, not a provider cache.

Is prompt caching the same as setting temperature to 0?

No, and the confusion is expensive. Temperature 0 makes sampling greedy — it reduces variance but still runs the full call. Prompt caching does not reduce variance at all; it removes redundant prefill work. They are orthogonal and usually used together.

Can I cache a prefix that contains retrieved documents?

Only if those documents are stable. A fixed policy corpus or product manual that changes weekly caches very well. Per-query retrieval results do not — they rewrite the prefix on every call and force a cold write each time. Structure it as static instructions and stable corpora in the cached prefix, per-query retrieval in the uncached suffix.

Do I still need prompt engineering if I use caching?

More, not less. Caching rewards prompts whose stable part is genuinely stable, which forces deliberate decisions about what belongs in instructions, what belongs in the variable turn, and what should never be interpolated at all. Our prompt engineering guide covers that discipline; caching simply makes the cost of getting it wrong visible in your usage numbers.

Conclusion

Prompt caching is the cheapest latency and cost win available to any workload with a large repeated prefix, and it asks for nothing but discipline: stable bytes first, variable bytes last, no volatile values above the breakpoint, and a metric that proves hits are actually happening. Put the template behind one gateway key so every caller shares the same warm prefix, watch the write-to-read ratio as your early warning that a prefix drifted, and the savings take care of themselves.

Start this week: instrument the usage fields above on your highest-volume endpoint and split the hit rate by template. If the prefix is long, stable, and reused but the hit rate is still low, you have just found a cost bug with a one-line fix.

Related reading

• Semantic Caching for AI APIs: Cut Latency and Cost by Up to 60%
• How to Reduce AI API Costs: A Practical Guide for Developers
• AI Prompt Engineering for Reliable API Responses
• Managing the Context Window: Truncation, Summarization, and Sliding Windows
• What Is the Model Context Protocol (MCP)? Connect Your AI to Real Tools
• How to Connect Cursor, Cline and Continue to a Custom AI API Endpoint
• Load Testing LLM Apps: Throughput, TTFT, and Concurrency