Prompt Caching Explained: Save Up to 90% on Input Tokens

Prompt caching is the single highest-leverage optimization available to most LLM applications, and it's also the most misunderstood. Here's what it actually does, how the economics work, and when it won't help you.

The core idea

Every time you call a model, the provider's infrastructure has to process (technically: run through attention layers and populate a KV cache) every token you send — including the parts that are identical to your last call. Prompt caching lets the provider store that processed state for a repeated prefix and skip re-doing the work on the next call, passing part of the savings on to you as a lower per-token price for the cached portion.

The critical constraint: caching only works on an exact-match prefix. If your system prompt is followed by a document that changes every call, only the system-prompt portion (the part before the first change) is cacheable — everything after the first differing token has to be processed fresh.

Cache writes vs. cache reads

Providers generally split caching into two priced operations:

• Cache write — the first time a prefix is seen (or after its cache entry expires), you pay to create the cache entry. This is often priced at a small premium over the standard input rate, not a discount.
• Cache read — every subsequent call that hits the same cached prefix within the TTL (time-to-live) window pays the discounted rate — commonly 80–90% off standard input pricing.

This means caching pays off when a prefix is reused enough times to amortize the write premium. A prefix used exactly once is more expensive than not caching at all; a prefix reused across hundreds of calls is dramatically cheaper.

The break-even math

Say the standard input rate is $I per 1M tokens, the cache-write rate is $1.25×I, and the cache-read rate is $0.1×I (a 90% discount — roughly what several providers offer today). For a prefix reused N times:

• Without caching: cost = N × I
• With caching: cost = (1 × 1.25×I) + ((N − 1) × 0.1×I)

Setting these equal and solving shows the break-even point arrives after roughly 2 reuses — by the third call on the same prefix, caching is already cheaper, and the gap widens fast after that. For any prefix you expect to reuse more than a handful of times (which describes almost every system prompt, few-shot example set, or repeatedly-referenced document), caching wins decisively.

What qualifies as "the same prefix"

This is where most implementations leak savings without realizing it. To keep a prefix cacheable across calls:

• Put stable content first. System prompt, tool/function definitions, and any fixed reference material should come before anything that varies (the user's actual message, retrieved chunks that differ per query).
• Keep it byte-identical. A single extra whitespace character, a timestamp injected into the system prompt, or a randomized few-shot example order will invalidate the cache silently — you'll pay full price and never know why.
• Watch the TTL. Cached prefixes expire after a provider-specific window (often minutes, sometimes longer with an explicit refresh). Bursty, infrequent traffic may fall outside the window between calls and never benefit.

Where caching helps most

• RAG and document Q&A — a large retrieved document or knowledge base chunk reused across a session or across many users asking about the same source.
• Agents and tool-use loops — the tool schema and system instructions are identical on every step of a multi-step agent loop, often the majority of the tokens in each call.
• Multi-turn chat — the accumulating conversation history is, by definition, a growing shared prefix from one turn to the next.

Where it doesn't help

• One-shot, highly varied prompts where no meaningful prefix repeats between calls.
• Prompts under the provider's minimum cacheable length (commonly a low four-figure token count) — caching typically has a floor below which it isn't offered at all.
• Extremely bursty traffic that falls outside the cache TTL between requests.

Practical takeaway

If your application has any repeated structure at all — and almost all production LLM apps do, whether it's a system prompt, a tool schema, or a knowledge base excerpt — restructure your requests so the stable part comes first and stays identical, then measure your actual cache-hit rate. Plug your estimated hit rate into the calculator, which models the input/output split explicitly, or see the full cost-optimization playbook for how caching fits alongside batching and model routing.