Prompt Caching Is the Cost Lever Most AI Teams Still Have Not Pulled

A RAG assistant prepends the same 40-page policy manual to every request. The manual never changes; the user question at the bottom does. The bill looks like the model is re-reading that manual from scratch on every call, because it is. The fix is not a missing feature. Prompt caching was in the API the whole time. The prompt was never structured to hit it, and nobody read the usage object to confirm.

That is the common failure. The discount is real and large: cache reads run at 0.1x the base input price on the Anthropic Claude API, a 90% reduction (Anthropic prompt caching docs), and DeepSeek bills cache hits at roughly 2% of the miss price (DeepSeek pricing). But those numbers are not a flag you flip. You earn them by laying out the prompt so the cache can match it, sizing the cached block above the model's minimum, and reading the usage object to confirm.

Why automatic caching still misses

The confusion starts with the word "automatic." On the OpenAI API, caching is on by default for prompts of 1,024 tokens or more, in 128-token increments, with no code changes and no extra fee (OpenAI prompt caching docs). DeepSeek's Context Caching on Disk is also on by default for everyone (DeepSeek KV cache docs). Gemini turns on implicit caching by default for 2.5 and newer models (Gemini caching docs).

"Automatic" means the provider will try. It does not mean your prompt is shaped to be cacheable. A cache entry is keyed on an exact, repeated prefix. Change one byte near the front of the prompt and the match breaks for everything after it. So the work is not "enable caching." The work is "build a prompt whose expensive, repeated part sits in a stable prefix that the matcher can find, and is long enough to qualify."

The three-token cost model you are actually billed on

Before touching layout, get the cost model right, because a poorly structured cached prompt can cost more than no caching at all. Billing splits your input tokens into three classes:

• Cache write — tokens written into the cache the first time. On Anthropic this carries a surcharge: 1.25x the base input price for the 5-minute TTL, 2x for the 1-hour TTL (Anthropic docs).
• Cache read — tokens served from a warm cache. This is the discounted class: 0.1x base on Anthropic, about 2% of the miss price on DeepSeek (DeepSeek pricing).
• Uncached input — tokens that were never cacheable or missed. Billed at full base price.

The Anthropic write surcharge is the part the comparison tables skip. You pay 1.25x to put the block in the cache once, then 0.1x every time you read it back. The break-even is immediate: a write plus a single read costs 1.25x + 0.1x = 1.35x across those two calls, versus 2.0x for the same two calls with no cache. You are net positive on the very first cache read, because each read saves 0.9x off full price and a single read more than covers the extra 0.25x you spent writing.

The only case the surcharge never recovers is a true one-shot: a prefix written and never read before the TTL expires. You pay 1.25x for a 1.0x job. Cache that prefix and you have spent the write premium for nothing.

Confirm the cache with the usage object

Do not trust that caching is working. Read it from the response. Every provider reports it, and this is your first observability signal.

Anthropic returns cache_creation_input_tokens (written) and cache_read_input_tokens (served), where total input equals cache_read plus cache_creation plus uncached input_tokens (Anthropic docs). OpenAI exposes cached_tokens under prompt_tokens_details (OpenAI docs). DeepSeek returns prompt_cache_hit_tokens and prompt_cache_miss_tokens (DeepSeek docs).

# Anthropic: first call writes the cache, second call should read it
"usage": {
  "input_tokens": 31,                    # the volatile user turn
  "cache_creation_input_tokens": 4096,   # written this call (surcharged)
  "cache_read_input_tokens": 0           # nothing warm yet
}

# Second identical-prefix call within the TTL:
"usage": {
  "input_tokens": 27,
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 4096        # served at 0.1x base
}

If your second call still shows cache_read_input_tokens: 0, the cache missed. Something in the prefix changed, the block was below the model's minimum, or the TTL expired.

Wire the read rate into a metric rather than eyeballing it once. Compute cache_read_input_tokens / (cache_read_input_tokens + cache_creation_input_tokens + input_tokens) per response and emit it as a gauge alongside latency and error metrics. For a high-volume call with a stable prefix, that ratio should sit near the static fraction of the prompt, often 0.8 or higher. Alert when the rolling average drops below a floor set from a known-good baseline. A read rate stuck at zero across identical-prefix calls means a prefix byte is changing on every request; diff the rendered prompt bytes between two calls to find it.

The cache-aware prompt layout every provider rewards

One layout rule covers every provider because they all match on a repeated prefix from the front of the prompt: order content from least volatile to most volatile.

1. Static system instructions: the role, the rules, the output contract. These rarely change.
2. Tool / function definitions: large JSON schemas that are identical across calls.
3. Reference documents: the RAG context, the policy manual, the API spec you keep re-sending.
4. Conversation history: grows but is append-only, so the older turns stay a stable prefix.
5. The volatile user turn: the one thing that changes every call. Always last.

The bug in the opening manual case is a layout inversion: a per-request timestamp or session ID injected near the top, ahead of the static manual. One volatile token at the front invalidates the cache for everything behind it. The fix is mechanical: move the static blocks to the front, push the variable token down into the user turn, and the read counter lights up on the next call.

Splitting breakpoints by change frequency

On Anthropic you place explicit markers. You get up to 4 cache_control breakpoints per request, and the system walks backward up to 20 blocks to find the longest matching cached prefix (Anthropic docs). Use the breakpoints to separate content by how often it changes:

"system": [
  { "type": "text", "text": "<static role + rules>",
    "cache_control": {"type": "ephemeral"} },          # breakpoint 1: ~never changes
  { "type": "text", "text": "<tool definitions JSON>",
    "cache_control": {"type": "ephemeral"} },          # breakpoint 2: changes on deploy
  { "type": "text", "text": "<RAG documents for this session>",
    "cache_control": {"type": "ephemeral", "ttl": "1h"} } # breakpoint 3: reused all session
]
# the user turn carries NO cache_control — it is the volatile tail

Mark the boundary after each block whose contents change on a different cadence. That way a tool-schema edit invalidates the tool block and everything after it, but the role block in front still serves from cache.

Per-provider differences that matter

The layout rule is universal. The activation contract is not, and routing through a gateway does not erase the difference.

Provider	Activation	Min tokens	Read discount	Write cost
Anthropic Claude API	Explicit cache_control	4,096 (Opus 4.x, Haiku 4.5); 2,048 (Sonnet 4.6); 1,024 (legacy Sonnet)	0.1x base	1.25x (5min) / 2x (1hr)
OpenAI API	Automatic	1,024 (128-token increments)	up to 90% off	no extra fee
Gemini	Implicit auto / explicit manual	2,048 (2.5 Flash/Pro)	reduced rate	+ storage charge (explicit)
DeepSeek	Automatic (disk)	best-effort	~2% of miss price	no extra fee

Anthropic is the only major provider where you opt in per request, and the only one that charges to write. Its minimum is also the highest and the most misread: current Opus 4.x and Haiku 4.5 will not cache a block under 4,096 tokens, Sonnet 4.6 needs 2,048, and only legacy Sonnet (4.5 and earlier) drops to 1,024 (Anthropic docs). A 1,500-token block that caches fine on legacy Sonnet silently writes nothing on Opus. OpenAI's cached-input discount runs up to 90% off on current flagship models; check the live numbers on the provider's own pricing page rather than a third-party table before building a budget on them. Gemini's explicit cache is the odd one out: it bills reduced-rate cached tokens plus a storage charge for the TTL duration, defaulting to 1 hour if you leave it unset (Gemini docs). DeepSeek's disk cache is the cheapest hit on the board, billing cache-hit input at roughly 2% of the miss price (DeepSeek pricing).

Where you run it changes the contract

The same model behaves differently by platform. Deploying Claude through Anthropic Bedrock can carry different token minimums than the first-party API, so a block that caches on the direct API may fall below the minimum on AWS. Running Gemini context caching through Google Vertex AI, or OpenAI through Azure OpenAI Service, gives you the same mechanics behind enterprise-deployment wrappers. Verify the minimums against the platform you actually deploy on, not the one in the headline docs.

What kills cache hits

When a team says "caching doesn't work," it is almost always one of these. Run this list against any prompt whose read counter sits at zero:

• A changed prefix byte. Matching is exact. A per-request timestamp, a shuffled JSON key order, a session ID, or a trailing whitespace edit near the front breaks the match for everything after it.
• Sub-minimum blocks. Below the model's floor, a block simply will not cache. On Anthropic that floor is 4,096 tokens for current Opus 4.x and Haiku 4.5, 2,048 for Sonnet 4.6, and 1,024 only for legacy Sonnet 4.5 and earlier (Anthropic docs); it is 1,024 on OpenAI (OpenAI docs) and 2,048 for Gemini 2.5 Flash/Pro (Gemini docs). A block sized for one tier writes nothing on a higher one.
• The 128-token alignment. OpenAI caches in 128-token increments, so the warm portion is the cached prefix rounded down to the nearest 128. A few stray tokens past a boundary are not free.
• Breakpoint placement. On Anthropic the markers define where a cacheable prefix can end. Put the only breakpoint after the volatile user turn and you cache nothing reusable. Markers belong after the static blocks.
• TTL expiry. OpenAI prefixes stay warm for 5 to 10 minutes of inactivity, up to an hour, and up to 24 hours on newer models (OpenAI docs). Anthropic's default is 5 minutes unless you ask for 1h. Slow traffic lets the cache go cold between calls and you pay the write again.
• Multi-provider routing. A gateway that load-balances the same model across providers can land your second call on a host with a cold cache.

The gateway gotcha in detail

Routing through OpenRouter passes provider caching through, but the contracts do not flatten. OpenAI, Gemini 2.5 implicit, and DeepSeek stay automatic, while Anthropic and Alibaba still require enabling caching per message via cache_control even through the gateway (OpenRouter caching docs). The subtler trap is routing itself: a cache is warm on the specific provider host that wrote it. If your next request lands elsewhere, it misses. OpenRouter handles this with sticky routing, keeping subsequent same-model requests on the same provider to keep the cache warm. If you run your own router, replicate that stickiness or your hit rate will look random.

This is also what a caching incident looks like on a dashboard. A clean prefix shows a hit-rate gauge that is high and flat. A provider-routing change, a deploy that edited a tool schema, or a load balancer that stopped honoring stickiness all show the same signature: the gauge falls off a cliff and then flaps, because some requests land on a warm host and the rest write cold. If you see that, do not chase latency. Pull two failing requests, diff the rendered prefixes, and confirm they were even served by the same provider host before looking anywhere else.

When not to cache at all

Caching is a cost lever, and a lever can be pulled the wrong way. Skip it when:

• The prompt is under the minimum. Below the model's floor, 4,096 tokens for current Opus and Haiku, 2,048 for Sonnet 4.6, 1,024 for OpenAI and legacy Sonnet, there is no cacheable block, so the effort is wasted.
• It is a one-shot call. No reuse means the Anthropic write surcharge never recovers. You pay 1.25x for a 1.0x job.
• The prefix is highly volatile. If the "static" front of your prompt changes most calls, you are paying write surcharges repeatedly and almost never reading back. That nets negative.
• Traffic is too sparse for the TTL. If calls arrive further apart than the cache lives, every call is a cold write. Either tighten the layout to justify the 1-hour TTL or do not cache.

The decision rule is one question: will this exact prefix be read back even once before the TTL expires? A single read already amortizes the write surcharge, so if the answer is yes, cache it and split breakpoints by change frequency. If no, leave it uncached and spend the engineering effort elsewhere.

A rollout runbook

Treat enabling caching like any production change. Roll it out behind a measurement, not a hope.

1. Profile the prompt. Measure what fraction of input tokens is static and repeated across calls. If most of the prompt is the volatile user turn, caching has little to grab.
2. Reorder. Move static system, tools, and reference docs to the front; push everything per-request to the tail. Strip volatile tokens (timestamps, request IDs) out of the prefix.
3. Check the block clears the minimum. Count the tokens in the cached prefix against the floor for the exact model you deploy: 4,096 for current Opus and Haiku 4.5, 2,048 for Sonnet 4.6, 1,024 for OpenAI. A block that clears the floor on staging but not in production caches on one and silently not the other.
4. Mark breakpoints by cadence (Anthropic only). One marker per change-frequency boundary, up to 4.
5. Pick the TTL. Default 5-minute for bursty traffic; 1-hour only when reuse within the hour clearly beats the 2x write cost.
6. Verify the usage object. Make a call, then an identical-prefix call, and confirm cache_read_input_tokens (or cached_tokens / prompt_cache_hit_tokens) goes positive on the second.
7. Pin a cache key if available. OpenAI accepts an optional prompt_cache_key that combines with the prefix hash to improve hit rates (OpenAI docs).
8. Alert on hit rate. Track cache_read over total input. A drop means a prefix change shipped, a deploy invalidated a block, or routing went unsticky.

Rolling back when a change tanks the hit rate

Caching is non-destructive, which makes rollback cheap. If a deploy drops the hit rate, you have two safe moves. The first is to revert the prefix change: restore the exact bytes of the previously-cached blocks and confirm cache_read_input_tokens recovers on the next identical-prefix call. The second, when you cannot revert immediately, is to disable caching outright. Drop the cache_control markers on Anthropic, or stop sending the cached prefix as a stable block. Because caching only reduces cost and never changes the model's output, turning it off is a safe instant rollback: you pay full input price until the prefix is fixed, but you ship nothing broken. Restore the markers once the read counter confirms the prefix is stable.

Take your highest-volume call, move every static block ahead of the user turn, size the cached prefix above the floor for your model, and make two identical-prefix calls. Read the usage object on the second. If the cache-read counter is above zero, ship it behind the hit-rate gauge and watch the ratio for the first week. If it is still zero, a prefix byte is changing up front, and that diff is the next thing to hunt down.