Semantic caching stores each AI response keyed by the embedding of its prompt, then answers a new request from the cache when cosine similarity to a stored prompt clears a threshold. Exact-match caches miss rephrased questions; semantic caches hit them. Teams running it on support, docs Q&A, and classification traffic typically see 30\u201360% fewer model calls with no measured quality drop.<\/p>\n\n\n\n
This guide covers the details that decide whether that number materialises: normalizing queries so paraphrases collapse, calibrating the threshold against your own embedding model, and choosing an invalidation strategy that survives a prompt edit.<\/p>\n\n\n\n
An exact cache keys on a hash of the fully-specified request: model ID, system prompt, message array, temperature, tool schema, response format. Two requests collide only if every byte matches. That makes it free, deterministic, and blind to meaning.<\/p>\n\n\n\n
Real traffic is full of near-duplicates that never hash equal. “How do I rotate an API key?”, “rotating an API key”, and “I need to change my API key \u2014 steps?” are three distinct hashes and one question. Support chat, docs Q&A, and IDE assistants generate paraphrase mass by design: an exact cache might serve 5\u201315% of that traffic, a semantic cache 30\u201350%, because the long tail is rephrasing, not repetition.<\/p>\n\n\n\n
Exact caching still earns its place, because retries and repeated eval runs produce byte-identical requests and a hash lookup costs microseconds. Run both \u2014 exact hash first, semantic second, model last<\/strong>.<\/p>\n\n\n\n The pipeline has seven steps. Two of them \u2014 normalization and scope \u2014 are the ones people get wrong.<\/p>\n\n\n\n Normalization is where hit rate is won.<\/strong> Strip anything that varies per request but not per answer: ISO timestamps, UUIDs, session IDs, a “user: alice” prefix, trailing whitespace, UI-added markdown wrappers. Do not stem or stopword-strip \u2014 aggressive normalization creates false collisions on short queries.<\/p>\n\n\n\n Scope stops you serving the wrong answer.<\/strong> A cached response is valid only for the same model, system prompt, and tenant. Put those in the vector metadata and filter on them at search time \u2014 never rely on similarity alone. A 0.97-similar prompt answered by a different model must be a miss.<\/p>\n\n\n\n Here is the whole thing in about forty lines, using an in-memory list and a linear cosine scan so the logic stays visible; swap Two details decide whether this is fast or slow. The embedding call is the entire hit-path cost<\/strong> \u2014 an ANN search over a million vectors is single-digit milliseconds, but a round trip to a hosted embedding endpoint is not. Use a small local model; a 384-dimensional sentence transformer is plenty for duplicate detection. Second, writes must be asynchronous<\/strong>: do the Cosine scores are not comparable across embedding models, dimensions, or text lengths \u2014 short queries produce noisier vectors and score systematically lower against longer cached prompts. A threshold copied from a blog post is a coin flip. Calibrate it in an afternoon:<\/p>\n\n\n\nHow a semantic cache works<\/h2>\n\n\n\n
request\n |\n v\n[1] normalize query strip volatile tokens (timestamps, request IDs, user names)\n |\n v\n[2] exact hash lookup in KV ---- hit ----> return cached response\n | miss\n v\n[3] embed the NORMALIZED query (must be the same embedding model that wrote the index)\n |\n v\n[4] ANN search: top-k nearest cached prompts, filtered by scope\n |\n v\n[5] best_score >= threshold AND scope matches (model + prompt_version + tenant)?\n | |\n yes no\n | |\n v v\nreturn cached response [6] call the model\n |\n v\n [7] async write: embedding + response + scope + TTL\n<\/code><\/pre>\n\n\n\nself.vectors<\/code> for a pgvector table in production and the methods are unchanged.<\/p>\n\n\n\nimport hashlib, time, numpy as np\n\ndef normalize(q: str) -> str:\n # collapse whitespace + case; strip volatile tokens in the real version\n return \" \".join(q.lower().split())\n\ndef cosine(a, b) -> float:\n a, b = np.asarray(a), np.asarray(b)\n return float(a @ b \/ (np.linalg.norm(a) * np.linalg.norm(b)))\n\nclass SemanticCache:\n \"\"\"Two-tier cache: exact hash first, then cosine similarity over embeddings.\"\"\"\n\n def __init__(self, embed, threshold=0.95, ttl=86_400, max_entries=50_000):\n self.embed = embed # str -> list[float]; SAME model for reads and writes\n self.threshold = threshold\n self.ttl = ttl\n self.max_entries = max_entries\n self.exact = {} # sha256 -> (expires_at, response)\n self.vectors = [] # (scope, embedding, expires_at, response)\n\n def _key(self, scope: str, query: str) -> str:\n raw = f\"{scope}\\x00{normalize(query)}\"\n return hashlib.sha256(raw.encode()).hexdigest()\n\n def get(self, scope: str, query: str):\n now = time.time()\n hit = self.exact.get(self._key(scope, query))\n if hit and hit[0] > now:\n return hit[1], \"exact\"\n\n qv = self.embed(normalize(query))\n best, best_score = None, 0.0\n for s, vec, exp, resp in self.vectors:\n if s != scope or exp <= now:\n continue # scope filter + TTL check, per candidate\n score = cosine(qv, vec)\n if score > best_score:\n best, best_score = resp, score\n\n if best is not None and best_score >= self.threshold:\n return best, \"semantic\"\n return None, f\"miss (best={best_score:.3f})\"\n\n def put(self, scope: str, query: str, response: str):\n now = time.time()\n self.exact[self._key(scope, query)] = (now + self.ttl, response)\n vec = self.embed(normalize(query))\n self.vectors.append((scope, vec, now + self.ttl, response))\n if len(self.vectors) > self.max_entries: # crude LRU; use the store's eviction\n self.vectors = self.vectors[-self.max_entries:]\n<\/code><\/pre>\n\n\n\nput<\/code> on a background task, and never make a miss slower than it would have been without a cache.<\/p>\n\n\n\nChoosing the similarity threshold<\/h2>\n\n\n\n
\n