Semantic caching stores each AI response keyed by the embedding of its prompt, then answers a new request from the cache when cosine similarity to a stored prompt clears a threshold. Exact-match caches miss rephrased questions; semantic caches hit them. Teams running it on support, docs Q&A, and classification traffic typically see 30–60% fewer model calls with no measured quality drop.
This guide covers the details that decide whether that number materialises: normalizing queries so paraphrases collapse, calibrating the threshold against your own embedding model, and choosing an invalidation strategy that survives a prompt edit.
Exact caching vs semantic caching
An exact cache keys on a hash of the fully-specified request: model ID, system prompt, message array, temperature, tool schema, response format. Two requests collide only if every byte matches. That makes it free, deterministic, and blind to meaning.
Real traffic is full of near-duplicates that never hash equal. “How do I rotate an API key?”, “rotating an API key”, and “I need to change my API key — steps?” are three distinct hashes and one question. Support chat, docs Q&A, and IDE assistants generate paraphrase mass by design: an exact cache might serve 5–15% of that traffic, a semantic cache 30–50%, because the long tail is rephrasing, not repetition.
Exact caching still earns its place, because retries and repeated eval runs produce byte-identical requests and a hash lookup costs microseconds. Run both — exact hash first, semantic second, model last.
How a semantic cache works
The pipeline has seven steps. Two of them — normalization and scope — are the ones people get wrong.
request
|
v
[1] normalize query strip volatile tokens (timestamps, request IDs, user names)
|
v
[2] exact hash lookup in KV ---- hit ----> return cached response
| miss
v
[3] embed the NORMALIZED query (must be the same embedding model that wrote the index)
|
v
[4] ANN search: top-k nearest cached prompts, filtered by scope
|
v
[5] best_score >= threshold AND scope matches (model + prompt_version + tenant)?
| |
yes no
| |
v v
return cached response [6] call the model
|
v
[7] async write: embedding + response + scope + TTL
Normalization is where hit rate is won. Strip anything that varies per request but not per answer: ISO timestamps, UUIDs, session IDs, a “user: alice” prefix, trailing whitespace, UI-added markdown wrappers. Do not stem or stopword-strip — aggressive normalization creates false collisions on short queries.
Scope stops you serving the wrong answer. A cached response is valid only for the same model, system prompt, and tenant. Put those in the vector metadata and filter on them at search time — never rely on similarity alone. A 0.97-similar prompt answered by a different model must be a miss.
Here is the whole thing in about forty lines, using an in-memory list and a linear cosine scan so the logic stays visible; swap self.vectors for a pgvector table in production and the methods are unchanged.
import hashlib, time, numpy as np
def normalize(q: str) -> str:
# collapse whitespace + case; strip volatile tokens in the real version
return " ".join(q.lower().split())
def cosine(a, b) -> float:
a, b = np.asarray(a), np.asarray(b)
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
class SemanticCache:
"""Two-tier cache: exact hash first, then cosine similarity over embeddings."""
def __init__(self, embed, threshold=0.95, ttl=86_400, max_entries=50_000):
self.embed = embed # str -> list[float]; SAME model for reads and writes
self.threshold = threshold
self.ttl = ttl
self.max_entries = max_entries
self.exact = {} # sha256 -> (expires_at, response)
self.vectors = [] # (scope, embedding, expires_at, response)
def _key(self, scope: str, query: str) -> str:
raw = f"{scope}\x00{normalize(query)}"
return hashlib.sha256(raw.encode()).hexdigest()
def get(self, scope: str, query: str):
now = time.time()
hit = self.exact.get(self._key(scope, query))
if hit and hit[0] > now:
return hit[1], "exact"
qv = self.embed(normalize(query))
best, best_score = None, 0.0
for s, vec, exp, resp in self.vectors:
if s != scope or exp <= now:
continue # scope filter + TTL check, per candidate
score = cosine(qv, vec)
if score > best_score:
best, best_score = resp, score
if best is not None and best_score >= self.threshold:
return best, "semantic"
return None, f"miss (best={best_score:.3f})"
def put(self, scope: str, query: str, response: str):
now = time.time()
self.exact[self._key(scope, query)] = (now + self.ttl, response)
vec = self.embed(normalize(query))
self.vectors.append((scope, vec, now + self.ttl, response))
if len(self.vectors) > self.max_entries: # crude LRU; use the store's eviction
self.vectors = self.vectors[-self.max_entries:]
Two details decide whether this is fast or slow. The embedding call is the entire hit-path cost — an ANN search over a million vectors is single-digit milliseconds, but a round trip to a hosted embedding endpoint is not. Use a small local model; a 384-dimensional sentence transformer is plenty for duplicate detection. Second, writes must be asynchronous: do the put on a background task, and never make a miss slower than it would have been without a cache.
Choosing the similarity threshold
Cosine scores are not comparable across embedding models, dimensions, or text lengths — short queries produce noisier vectors and score systematically lower against longer cached prompts. A threshold copied from a blog post is a coin flip. Calibrate it in an afternoon:
- Sample 200–500 real (new query, cached prompt) pairs from your logs, over-weighting suspected duplicates.
- Label each pair same question or different question. This is the only expensive step.
- Sweep the threshold from 0.80 to 0.99 and, at each step, compute precision (share of hits that were genuinely the same question) and hit rate.
- Pick the lowest threshold whose precision clears your tolerance. Precision is the dial; hit rate is the reward.
| Threshold | Use it for | Typical hit rate | Risk |
|---|---|---|---|
| 0.98–1.00 | Code generation, numeric output, legal or medical text — anything where a wrong answer is expensive | Very low (5–10%) | Near zero; behaves almost like an exact cache |
| 0.95–0.98 | Technical Q&A, API docs, code explanation. The safe production default | 15–30% | Occasional miss on aggressive paraphrases |
| 0.90–0.95 | Support chat, FAQ deflection, summarization of similar documents, intent classification | 30–50% | Low; needs a false-hit review loop |
| 0.85–0.90 | High-volume templated tasks (tagging, routing, sentiment) where a slightly off answer is cheap to correct | 45–65% | Moderate — audit weekly |
| Below 0.85 | Almost nothing | High but meaningless | Contradictory answers, inconsistent UX, silent correctness bugs |
Three refinements matter more than the number. Set the threshold per route, not globally — classification and code generation have different error tolerances and different score distributions. Add a margin rule: if the top two candidates both clear the threshold but sit within 0.01 of each other, treat it as a miss, because the query is ambiguous between two cached answers. And require a higher threshold for short queries, since below roughly five tokens the embedding cannot separate “reset password” from “reset PIN”.
Temperature is the last piece. At temperature 0 a cached response is exactly what the model would have produced; at 0.7 it is one sample from a distribution, so a hit and a miss phrase the same question differently. For how prompts become vectors, see embeddings and RAG.
Storage and TTL
You need two stores, not one. The exact tier is a plain key-value lookup; the semantic tier is an approximate-nearest-neighbour index with metadata filtering. They scale completely differently.
| Layer | Good default | Reach for something heavier when |
|---|---|---|
| Exact KV | Redis or your existing cache, one TTL per key | Almost never — this tier is trivially cheap |
| Vector index | pgvector, if you already run Postgres and hold under a few million entries | You need single-digit-millisecond ANN at high query rates, horizontal scale, or native metadata filtering over hundreds of millions of vectors |
Budget the memory before you switch it on. A 1536-dimensional float32 vector is roughly 6 KB, so one million entries is about 6 GB of index before overhead — the number that turns a cost-saving feature into a line item. Three levers cut that by an order of magnitude without hurting duplicate detection: float16 storage, matryoshka truncation to the first 256–512 dimensions, or binary quantization with a float32 rescoring pass. A 384-dimensional model often beats a 1536-dimensional one on memory and latency at equal precision.
TTL should be a function of how volatile the answer is, not one global constant — otherwise entries never stop accumulating, and a semantic index with a 30-day TTL under real traffic grows without bound.
- Model facts, pricing pages, policy text: 1–6 hours. These change without warning, and a stale answer is actively wrong rather than merely old.
- General how-to and conceptual explanations: 7–30 days. Stable by nature; this is where the savings live.
- Product documentation: tie the TTL to your docs deploy rather than the clock — a version tag in the scope beats a timer.
- Anything derived from live data (inventory, order state, market data): do not cache, or set the TTL below the source’s refresh interval.
Always pair TTL with a hard size cap and LRU or LFU eviction, whichever triggers first. TTL bounds staleness; the size cap bounds your memory bill. Configuring only one of them is how semantic caches turn into incidents.
Invalidation strategies
Time-based TTL is the baseline and you should always have it — but relying on it alone means either stale answers or a cache that expires before it pays for itself. Four sharper mechanisms, in rough order of value:
- System-prompt hash in the scope key. Store a hash of the system prompt as a metadata field you filter on. The moment you edit the prompt, every entry written under the old hash becomes unreachable — automatically, with no delete job. Prompt edits are the most common cause of stale answers, and this costs one field.
- Version tags as a namespace. Put
prompt_version,model_id,tool_schema_version, andcorpus_versionon every entry and filter on all of them. Bumping any tag is an O(1) global invalidation: stop matching the old namespace and let TTL reap the orphans — no delete storm, no downtime. - Semantic delete. To retract a single fact, embed it and delete entries whose prompt embedding sits within a tight radius (cosine above roughly 0.97) and whose scope matches. Keep the radius tight — a loose one removes legitimate neighbours along with the target. This is also your deletion-request mechanism: store a subject identifier in metadata and delete by filter.
- Negative and refusal caching. Refusals are the most expensive misses to repeat and the most likely to be false negatives. Cache them with a much shorter TTL — minutes rather than days.
Never invalidate by string-matching on prompt text. It breaks on the first rephrase — the exact problem the semantic cache exists to solve.
When NOT to cache
Semantic caching is a correctness trade, and for some workloads the trade is bad. Skip it when any of these apply:
- Creative or high-temperature generation. Brainstorming, copy variants, “give me five names” — a cached answer defeats the request, and users notice when the second attempt is character-identical to the first.
- Per-user or personalized output. Anything conditioned on conversation history, a user profile, or account state. Caching across users leaks data; caching per user yields a hit rate near zero.
- Real-time data. Prices, availability, status — anything whose refresh interval is shorter than your TTL. Exclude it, or set the TTL below the refresh interval.
- Agentic and tool-calling loops. The same prompt legitimately produces different answers when tool results differ. Cache the tool result instead, or fold a hash of the tool state into the scope key.
- Prompts dominated by a unique payload. If every request embeds a document the user just uploaded, the embedding is mostly document and you will never hit. Cache at the sub-question level instead.
- Anything cross-tenant. If a near-duplicate could return another customer’s data, the feature is a security bug, not an optimization.
One exception worth knowing: streaming works fine with semantic caching. Cache the fully assembled text, then on a hit replay it as synthetic SSE chunks on a short timer. Client code does not change and perceived latency collapses. Our streaming and SSE guide covers the chunk format if you need to match it exactly.
Measuring impact
Four metrics, reported separately for exact hits, semantic hits, and misses. Averaging them hides the entire effect.
- Hit rate — hits ÷ total requests, split by tier. The headline number.
- p50 and p95 latency per tier. Report hit latency and miss latency as separate series. The visible p95 improves only in proportion to hit rate: a 40% hit rate with a 10× faster hit path yields roughly a 3× p95 improvement, not 10×.
- Effective cost per request — (miss rate × unit inference cost) + (embedding cost + amortized index cost). The embedding step is two to three orders of magnitude cheaper per token than generation, so this should be dominated by the miss rate.
- False-hit rate. Sample a few hundred cache hits per week and judge whether the cached answer actually answered the new question. Target under 1–2% — this is the metric that keeps your threshold honest.
| Metric | Before caching | After (0.93 threshold, FAQ workload) | Change |
|---|---|---|---|
| Model calls per 100k requests | 100,000 | 42,000 | −58% |
| Inference spend (relative) | 1.00× | 0.44× | −56% |
| p95 end-to-end latency | 1.00× | 0.38× | −62% |
| p50 end-to-end latency | 1.00× | 0.35× | −65% |
| Embedding + index cost | — | +0.03× | +3% |
| Measured false-hit rate | — | 0.7% | — |
| Throttling events | 1.00× | 0.42× | −58% |
Those figures are illustrative for a paraphrasing-heavy support workload. Your numbers depend almost entirely on paraphrase density, so measure it before you commit: cluster one day of real prompts by embedding and look at the size of the clusters above your threshold. That distribution is your expected hit rate. If your traffic is mostly unique long-context requests, do not build this.
One architectural note changes the economics: run the cache in the gateway rather than inside each application. A gateway sees every request from every service, so one index is shared across all of them — which multiplies the hit rate without multiplying the infrastructure. It is also the natural seam for adjacent edge concerns: fallback routing when a provider throttles, and the retry policy that turns a rate limit into a queued request instead of a failed one. A relay such as qoraapi.com, which fronts many models behind one OpenAI-compatible endpoint, is exactly that seam — the request already passes through one process, so the cache is a layer rather than a refactor. For the wider set of cost levers, see our guide on how to reduce AI API costs.
Frequently asked questions
Does semantic caching reduce answer quality?
Not if the threshold is calibrated and the false-hit rate is measured. The cache changes which questions are answered from memory, not which model answers them — a hit returns a response the same model already produced for a question your labelers judged identical. The failure mode is a threshold set too low and never audited.
Can I use semantic caching with streaming responses?
Yes. Cache the final assembled text and replay it as synthetic SSE chunks on a short interval. Do not cache a partial stream — a half-generated answer is not a reusable artifact, and a client that disconnects mid-stream would poison the entry.
How much does the embedding step cost?
Roughly two to three orders of magnitude less per token than generation, so it is almost never the reason a cache stops paying for itself. The real risk is latency: a hosted embedding round trip adds tens of milliseconds to every request, including misses.
Do I need a dedicated vector database?
Usually not at first. pgvector handles millions of entries with metadata filtering and keeps you on infrastructure you already operate, which matters more than ANN benchmarks while you are still calibrating a threshold. Move to a dedicated vector store when you need single-digit-millisecond search at high query rates.
Conclusion
Semantic caching is not a clever trick — it is a threshold you calibrated, a scope you enforce, and a TTL you chose deliberately. Normalize before embedding, run the exact lookup ahead of the vector search, filter every candidate by model and prompt version, and pick a threshold from your own labeled pairs. Then let the false-hit rate — not the hit rate — decide when to stop tuning.
The payoff is workload-dependent: paraphrase-heavy traffic sees 30–60% fewer model calls and a proportionally faster p95, while unique long-context traffic sees almost nothing and should not pay for an index.
Leave a Reply