A production LLM application rarely receives a question nobody has asked before. Support assistants and retrieval pipelines field the same intent thousands of times a day, phrased differently each time.
Most stacks treat those repeats as fresh, fully billed requests. But the cost piles up in two separate places, and those two places have different fixes.
This guide separates the two caching layers: one makes the call cheaper, the other removes the call entirely. Which one helps which workload, how much they save when stacked in the right order, and where to stop are below.
Two caches, two kinds of saving
The difference fits in one sentence: a prefix-cache hit is a cheaper generation call, not an avoided one. A semantic-cache hit means the model is never reached at all.
That distinction decides the whole cost calculation. The first layer reduces latency and the number of tokens processed; the second removes the request itself.
- Prefix cache: stores the computed form of the prompt's fixed beginning; the model still runs.
- Prefix-aware routing: sends requests sharing a beginning to the same machine so that cache actually warms up.
- Semantic cache: returns the stored answer when a question matching in meaning has been answered before.
- All three together: the layers do not replace each other, they run in sequence.
- Order matters: when the semantic layer misses, the request falls through to the prefix cache anyway.
What prefix caching does
The prompt sent to a model in an LLM application usually has two parts: a fixed section that sets up context and a variable section carrying the actual user input.
| Part | Example | Typical size |
|---|---|---|
| Fixed section | Instructions, policy documents, conversation history | 3,000 tokens |
| Variable section | The question the user typed | 50 tokens |
| Result | The same 3,000 tokens processed again on every request | per request |
It is easy to make concrete with an example. In a customer service bot every request starts with the same block of text: "You are a support agent for AnyCompany. Here are our policies…" followed by whatever the customer typed. If those instructions run to 3,000 tokens, then across hundreds of requests the model is processing the same 3,000 tokens over and over.
Serving frameworks such as vLLM and TensorRT-LLM have a solution. They cache the computed key-value pairs for prompt prefixes they have seen before. When the same beginning appears in a new request, the model reuses that computation and only processes the new tokens at the end.
This is called prefix caching and it can cut time-to-first-token significantly. On a single machine it works as expected.
The scale problem: a fleet spreads the cache thin
Once you move past one machine and put a fleet behind an endpoint, the mechanism breaks. Requests get distributed across all of them.
- The same 3,000-token prefix lands on instance A for one request, B for the next, then C.
- No instance sees that prefix often enough to build a reliable cache.
- The caching feature is switched on, but the routing layer spreads requests too thinly for it to help.
So the problem is not in the cache but in how requests are distributed. The feature works; it just never gets a chance to warm up. And it gets worse as the fleet grows: the more instances there are, the lower the odds of catching the same prefix on the same machine.
Prefix-aware routing
Amazon SageMaker Inference added a strategy for exactly this point, called prefix-aware routing. It looks at the beginning of each request arriving at the endpoint and consistently sends requests with the same beginning to the same instance.
- If ten requests share a prefix, all ten go to the same machine and that machine's cache stays warm.
- Different beginnings spread across different instances, so load balance is preserved.
- You do not need to tag requests or manage affinity yourself.
- In a Llama 3.1 70B benchmark, KV cache hit rates went from roughly 25% to over 80%.
In the same benchmark P50 time-to-first-token fell by up to 77% and throughput rose by up to 16%. Those are notable numbers, but we are still talking about making a generation call cheaper. For a team running vLLM on its own infrastructure the lesson is the same: switching the cache on is not enough, the routing has to be built for it too.
The gain here is real but its ceiling is clear. The request still reaches the model, new tokens still get processed, and the full answer still gets decoded. A prefix-cache hit shrinks the bill rather than zeroing it.
Semantic caching: never making the call
The second layer answers a different question. Consider three requests arriving at a support assistant: "Can I get a refund after buying the monthly plan?", "Is the monthly subscription refundable?", "Can I cancel the plan and get my money back?"
The wording differs but the question and the answer are identical. Without a semantic cache each of those phrasings triggers a complete generation: input tokens processed, output tokens decoded, the user waiting.
Redis offers this layer as a managed service called LangCache. It sits between the application and the model, matches incoming prompts against previously answered ones by meaning rather than by text, and returns the stored response when a close enough match exists. The company reports API cost savings of up to 90% and cache-hit responses up to 15 times faster.
A two-call loop
The architecture is simple and comes down to two calls on the application side.
- Before invoking the model, the prompt goes to the search endpoint; the service generates an embedding and runs a vector search over stored entries.
- If a semantically similar entry clears the configured threshold, the stored response is returned and the model is never called.
- On a miss, the application calls its chosen model as usual.
- The prompt and the new response are then written to the entries endpoint, joining the pool for future matches.
Embedding generation is handled by the service; it can work with default models or you can bring your own. The service is currently in public preview on Redis Cloud, accessed through a REST API, with Python and JavaScript SDKs. Being a preview, it is noted that behaviour may change.
What to measure
The numbers that show whether each layer is working are different too. Judging one by the other's metric makes the gain look larger or smaller than it is.
- For the prefix cache: KV cache hit rate and P50 time-to-first-token. A low hit rate means you have a routing problem.
- For the semantic cache: hit rate and the number of calls avoided. The line on the invoice depends directly on the second.
- Shared metric: total cost per request. Watch how that single number moves as you add layers.
- The overlooked item: the cost of generating embeddings. At a low hit rate it can eat the saving.
Measuring layer by layer matters as well. Turn both on in the same week and you cannot tell which gain came from where, which leaves the next adjustment to guesswork.
Where to set the similarity threshold
The one real risk of a semantic cache is a false hit: returning an old answer to a question that only looks similar. The similarity threshold is what manages that risk.
Set it high and the hit rate drops along with the saving; set it low and the chance of a wrong answer rises. Alongside it the service offers TTLs, eviction policies and adaptive controls that tune precision and recall. In practice the right path is to start high and walk it down while reading the recorded hits by hand.
Separating question types helps too when tuning. Policy and definition questions tolerate a wide threshold because the answer does not change over time. For anything tied to a person or a moment — account status, a price — the semantic cache should be off from the start; there the right answer is not yesterday's answer.
Which one for which workload
The two layers pay off on different patterns. Prefix caching helps any application with a long, fixed system prompt, even when no two questions resemble each other.
A semantic cache only produces value if questions repeat. Support assistants, FAQs and retrieval-based search fit that description. In a code generation tool where every request is unique you will get almost no hits, and the added embedding cost turns into a loss.
A short note for non-English workloads. Semantic matching depends on how well the embedding model represents the language, and multilingual models may not draw distinctions in Turkish as sharply as in English. Carrying over a threshold you tuned on English can cause quiet false hits.
Where to stop
Measure first. Building a semantic cache without first working out from your logs how many incoming requests are semantic repeats means picking the solution before the problem.
Order matters too: first tidy the fixed prompt so it can benefit from prefix caching, then set routing up for it if the fleet is large, and only then add the semantic layer. Done backwards, the second layer hides an unsolved problem in the first and the bill does not fall as far as you expected. A cache preserves a badly built prompt or a needlessly long context just as faithfully; the real saving usually starts with shortening that text.