Semantic Caching for LLMs: Cut Cost & Latency

Why a normal cache is not enough

A traditional cache keys on exact input: ask the same question twice and the second is free. But users never phrase things identically. "How do I reset my password?" and "I forgot my password, what now?" mean the same thing, yet an exact-match cache misses entirely. Semantic caching fixes this by matching on *meaning*, not characters — and for LLM apps it can dramatically cut both cost and latency.

How semantic caching works

1. Embed the query into a vector.
2. Search a cache of previous query vectors for a close match.
3. If the nearest match is above a similarity threshold, return its stored answer — no LLM call.
4. Otherwise, call the model, then store the new query vector and answer for next time.

{
  "query": "I forgot my password",
  "query_vector": [0.11, -0.04, 0.92],
  "answer": "Click 'Forgot password' on the login page to get a reset link.",
  "model": "claude-haiku-4-5",
  "created": "2026-06-24T10:00:00Z",
  "hits": 7
}

Each cache entry is a JSON record holding the original query, its embedding, the answer, and metadata for eviction and analytics.

Tuning the similarity threshold

The threshold is the critical dial:

• Too low → false hits: the cache returns an answer to a question that only *sounds* similar. Wrong and embarrassing.
• Too high → few hits: you rarely reuse anything and lose the savings.

Start conservative (high threshold), measure false-hit rate on a sample, and loosen carefully. For factual support FAQs you can be aggressive; for nuanced or personalized queries, be strict or skip caching entirely.

What to cache — and what not to

Add a TTL so answers expire, and include context (user, locale) in the cache key when responses legitimately differ.

The payoff

• Cost: a cache hit costs an embedding lookup instead of a full generation — often an order of magnitude cheaper.
• Latency: returning a stored answer is milliseconds versus seconds.
• Throughput: fewer model calls means you stay under rate limits during spikes.

Even a modest hit rate pays for the caching infrastructure quickly on a high-traffic app.

Implementation notes

• Use the same embedding model for cache writes and lookups.
• Store entries in a vector database so similarity search is fast at scale.
• Track hits and last-used time to evict cold entries.
• Log near-misses to tune the threshold over time.