Original URL: https://getagenttools.com/blog/prompt-caching-explained/ # Prompt caching, explained by someone who actually pays the bill 23 Sep 2026Marek. - llm-cost - prompt-caching - claude-api Prompt caching saves you money only in one situation: the same static chunk of context shows up again in your next call, before the cache has expired. Everything else you hear about it, that it makes responses faster, that you should just flip it on everywhere, only holds up when that one condition is met. I did not learn this from a benchmark. I learned it from a checklist I run every time I design something that calls a model more than once, because it catches a cost problem before the monthly bill does. Here is what this covers: 1. the mechanics: KV pairs, prefill, and why caching is not the same as caching a full answer; 2. what is actually worth caching in a real setup; 3. when caching does nothing for your bill, and can occasionally cost you more; 4. the one-line check I use before I turn it on. ## What prompt caching actually is Start with what the term does not mean. Prompt caching is not the same as caching a full response, sometimes called output caching. Output caching stores the finished answer for an identical (or near identical) request and skips calling the model at all the next time that exact question comes in. It behaves like a lookup table: if the request matches, you get the old answer back, and you pay nothing for a new generation. Prompt caching works earlier in the pipeline, and it solves a different problem. When a model processes a prompt, it runs a step usually called prefill: for every token in that prompt, at every layer of the transformer, it computes a pair of vectors, the key and the value, that the attention mechanism needs to relate tokens to each other. This is the computationally expensive part of serving a request, and it happens before the model writes a single word of output. Prompt caching saves those key-value pairs, the KV pairs, for the part of the prompt that stays the same across calls. The next request that starts with that identical prefix does not repeat the prefill work for it. The model picks up from the saved state and only computes fresh KV pairs for whatever is new, usually a short instruction or question tacked onto the end. It still calls the model. It still generates a fresh answer. It just skips redoing the expensive part of processing content it has already seen. The distinction matters because the two techniques solve different failure modes. Output caching helps when many callers ask the literal same question. Prompt caching helps when different questions keep landing on the same static setup, which in practice is the far more common shape of real usage. A support tool that answers different user questions against the same product documentation cannot use output caching, because the questions differ every time. It is a strong candidate for prompt caching, because the documentation does not change at all. ## The mechanics, one level down It helps to picture what a model is doing across a session, not just a single call. Every layer of the transformer needs the key and value vectors for every prior token so it can decide how much attention to pay to each one. Computing all of those vectors for a long document is not a rounding error. It is the majority of the work in a request that includes a lot of input text and a short answer. A cache does not remove that cost. It pays it once instead of on every call. The first call that includes a new static prefix, such as a system prompt or a set of tool definitions, still has to pay the full prefill cost and store the resulting KV pairs. Every call after that, as long as it starts with the exact same prefix, reuses the stored state instead of recomputing it. Only the new part of the prompt, the part that was not there before, gets processed fresh: ```text call 1: [system prompt + tool definitions] + [user question A] -> prefill runs on everything -> KV pairs for the static part get cached call 2: [system prompt + tool definitions] + [user question B] -> cached KV pairs reused for the static part -> prefill only runs on question B ``` This is also why prompt caching is not permanent. The cached state sticks around only for a limited window, and how long that window is, and whether it can be extended, depends entirely on the provider and sometimes the plan. Once the window closes, the next call pays full price again and re-establishes the cache. I treat the actual duration as something to check in the docs for whichever API I am using, not something that carries over from a different provider or a different model. ## Cache it or skip it Before going deeper, here is the short version I actually use when deciding whether to bother: | Situation | Worth caching? | | --- | --- | | Long system prompt reused within the cache lifetime | Yes | | Tool or function schema sent with every request | Yes | | Long static document queried more than once | Yes | | Short prompt with little static content | Usually not | | Context rebuilt from live or per-user data on every call | No | | Same static prefix, calls further apart than the cache lifetime | No | The rest of this piece is just the reasoning behind that table. ## What is actually worth caching The best candidates for caching share one property: they do not change between calls, and they get reused a lot. In practice that narrows things down fast. A system prompt qualifies immediately in most agent setups. If every call to your agent starts with the same long, carefully written instructions, you are paying to reprocess that prompt from scratch on every single call unless it is cached. Tool and function definitions are just as good a fit. An agent with a dozen defined tools sends the full schema for each one on every call, and that schema rarely changes between one user turn and the next. [Claude Code](https://getagenttools.com/tools/claude-code/) and harnesses like it resend that same tool list and system prompt on every step of a multi-step task, not just once at the start, the same repeated load that fills up a long [context window](https://getagenttools.com/blog/claude-code-context-window/) turn after turn. That is static, repeated content sitting at the front of the request, which is close to the ideal shape for a cache. Large static documents round out the list, and they are probably where the savings are easiest to see. Paste a long report or a contract into the context and ask several different questions about it in the same session. The document is the expensive, unchanging part. The questions change. Cache the document, and every follow-up question pays full price only for itself, while the document is billed at the much lower cache-read rate. What ties these together is repetition, once a prefix is long enough to cache at all. A system prompt that clears the provider's minimum and gets reused thousands of times a day is a better caching candidate than a long document you only ever query once. ## When caching does nothing for your bill Caching only pays off when a shared prefix comes back before the cache expires. Plenty of real workloads do not work that way. A short prompt is the simplest case. When the whole request is short, there usually is not enough shared, reusable content to make caching worth the overhead of managing it: writing to the cache and checking whether a later call matches it costs attention, and, on the providers that charge one, a small price premium of its own. Where exactly the break-even point sits depends on how a given provider prices a cache write and a cache read against a normal token, and that is not a fixed number across the industry, so I check the current numbers for whichever API I am using instead of assuming a cutoff that applied somewhere else still applies here. A prompt that is different on every call is the case people get wrong more often. If you are building the prompt fresh each time from live data, a current price, a fresh search result, today's log lines, whatever is specific to this one user, there is no repeated prefix for the cache to catch. Every call looks new to the cache, so every call pays to write one and never gets to read from it. That second case is also where caching can quietly cost you more, not less. On providers that charge for cache writes, such as the Claude API, a write costs a small premium over a normal input [token](https://getagenttools.com/blog/what-is-a-token/), while a read costs well below it. If your requests never repeat, you keep paying the write premium and never collect the read discount. Turning caching on by default, without checking whether your context actually repeats, is how a cost optimization turns into a cost increase. The third case is timing. A cache lives for minutes by default, so a static prefix that shows up a few times a day has expired before the next call arrives. Every call starts cold, and on a provider that charges for writes, every call pays that premium for nothing. ## The cost rule I actually use I have a short cost checklist I go through before I build anything that calls a model more than once. One line on it asks whether prompt caching makes sense at all. The line right above it asks how many times a day the thing runs, and the caching answer needs both: a repeated block of more than about a thousand tokens, and calls that come often enough to find it still cached. Most providers set a minimum cacheable prefix, on the order of a thousand tokens, and below it the call is quietly not cached at all, so the length half of that question has a real floor, even though the exact number moves by provider and model. Once caching is on, I do not take my own word for it, either: usage counters report how many tokens on a call were read from cache. If that counter still sits at zero after several calls that are supposed to share the same prefix, either something in the prefix is changing without me noticing or the calls are too far apart for the cache to survive between them. On the Claude API, where writes have their own counter, a zero there as well means nothing was cached in the first place, usually because the prefix is under the minimum or was never marked. On an API that caches only what you mark, a first draft that never marks the static prefix pays full price for it on every call in the loop. The same tool definitions and the same long system instructions get recomputed from scratch on every call, inside a loop that can fire dozens of times, because nothing marks them as something to reuse. Nobody does it on purpose. The prefix works on the first call, so nobody goes back to ask whether the tenth call needs to pay for it again. The fix, once someone actually checks for it, usually takes a few lines: mark the static block and keep it identical across calls. The part that takes longer is remembering to check in the first place, which is the entire reason it lives on a checklist instead of in my head. That line already changed one decision for me. A mail pipeline I ran this year put the same block of static context, about 12,000 tokens, in front of every reply it drafted, but it handled about one and a half emails a day and the cache lived five minutes. Caching there would have meant paying the write premium on every draft and almost never reading it back, so I left it off. Compare caching options using [cost per successfully completed task](https://getagenttools.com/blog/ai-agent-cost/), including retries and failed runs rather than only the discounted token line. ## Before you flip the switch Prompt caching is not a setting you turn on once and forget. It is a bet that your next call will start the same way your last one did, and arrive before the cache expires. When that bet is good, caching is close to a free win: lower latency and lower cost, without changing the quality of the answer. When that bet is bad, you are adding a moving part to your system for no return, and paying a small premium for the privilege. Before you reach for it, look at your own traffic the way you would look at any other cost decision. Pull a sample of real requests. Check how much of each one is identical to the one before it, and how much time passed in between. If a large, static block keeps showing up at the front of your prompts before the cache expires, cache it. If it does not, spend your engineering time somewhere else. ## Frequently asked questions ### What is prompt caching? Prompt caching is a technique for reusing the internal state a model computed for the static part of a previous prompt, so a later call that starts with the same content does not have to process it again. It only helps when a prefix repeats across calls that arrive before the cache expires, and it is not the same as caching a full response. ### What is LLM cost optimization? LLM cost optimization is the general practice of reducing what it costs to run large language models in production without hurting the quality of the output. It covers choices like picking the right model size for a task, trimming unnecessary context, and reusing static context through mechanisms like prompt caching. ### What is a KV cache in LLMs? A KV cache holds the key and value vectors a transformer has already computed for every token, at every layer. While the model writes an answer token by token, it reuses them instead of recomputing them for everything before the current token. Prompt caching extends the same idea across requests: the vectors for a shared prefix are kept after one request ends, so the next request that starts the same way skips that work. ### How does prompt caching work with the Claude API? With the Claude API, you turn caching on with a `cache_control` field: on the block where the static part of the request ends, such as the end of the system prompt or of the tool definitions, or once at the top level, where the API moves the cache point forward as the conversation grows. On a later call within the cache lifetime, if everything up to that point is identical, the API reuses the stored state and processes only what comes after it. A cache write is billed above a normal input token, a cache read well below it. Put this into practice [See the tool list](https://getagenttools.com/tools/)[Train your team](https://getagenttools.com/#training)