Prompt caching, explained.
September 26, 2026
Prompt caching is the highest-ROI optimization in the playbook because it's provider-native, requires almost no code change, and discounts the most expensive token type on your bill: long, repeated input. Done right, you cut 30–60% off cache-eligible endpoints in a week.
How it works (in one paragraph)
The model's KV cache ; the intermediate state computed when reading your prompt ; is normally thrown away after each request. With prompt caching, the provider keeps that state for a few minutes and reuses it when your next request shares an identical prefix. You pay a small write fee the first time and a steeply discounted read fee on every subsequent hit. The output is identical to an uncached call.
What each provider charges
| Provider cache | Mechanics |
|---|---|
| OpenAI · cache-read | ~50% off input price · 5–10 min TTL · auto on prompts ≥ 1024 tokens |
| Anthropic · ephemeral cache | ~90% off input · 5 min TTL · 25% write premium · explicit `cache_control` block |
| Anthropic · extended cache | ~90% off input · 1 hour TTL · 100% write premium · explicit beta header |
| AWS Bedrock (Anthropic on Bedrock) | matches Anthropic terms · region availability varies |
| Google Vertex (Gemini) | context cache · per-token-per-hour storage fee + ~75% read discount · explicit cache resource |
How to structure prompts so the cache hits
- Static content first, dynamic content last. The cache matches a prefix. Anything before the first byte of difference is cacheable; everything after is not.
- Move tools, system prompt, and few-shot examples into the prefix. These are the largest static blocks in most production prompts.
- Pin the user message, retrieved docs, and timestamps to the suffix. They change per request anyway.
- Don't interpolate the date into the system prompt. One unstable byte per day kills a year of cache hits.
- Anthropic-only: add `cache_control: {type: "ephemeral"}` to the last block you want cached. Up to 4 break-points.
Common mistakes that silently disable caching
- Trailing whitespace from a templating engine breaking byte-equality.
- Random tool order when serializing a `tools` array ; sort it.
- JSON key order drift from a non-deterministic serializer.
- User ID or session ID embedded in the system prompt for "personalization." Move it to the user message.
- Streaming tools that re-issue the full conversation instead of reusing the conversation prefix. Check your SDK.
The cache-read token baseline trap
Cache-read tokens are billed and reported separately from input tokens on Anthropic. Many teams set their pre-engagement baseline by summing all input-tokens lines and miss the cache-read line entirely ; then "savings" mysteriously appear that are really just a column shift. Always reconcile against the raw provider invoice with cache-read as a distinct line item. We wrote up the trap in detail: cache-read tokens: the baseline trap.
When prompt caching doesn't help
- Single-shot prompts with no shared prefix between users.
- Hot reloads of system prompt on every deploy. Pin it.
- Prompts under the minimum token threshold (OpenAI: 1024 tokens; Anthropic: 1024 for Sonnet, 2048 for Haiku, 1024 for Opus).
- Workloads bursting then idling for hours. Use Anthropic extended (1-hr) cache or accept the cold-start cost.
Related
- Cache invalidation cost: the hidden line item
- Full LLM cost optimization guide
- Cache-read tokens: the baseline trap
- Semantic caching: when it works and when it doesn't
FAQ
How does prompt caching reduce LLM costs?
Prompt caching reuses the model state for repeated prompt prefixes instead of recomputing the full input every time. The provider charges a write fee when the prefix is first cached, then a discounted read fee on later matching requests. The output is identical to an uncached call, but long repeated input becomes much cheaper.
Which parts of a prompt should be cached?
Put static content first and dynamic content last because the cache matches a prefix. System prompts, tool definitions, and few-shot examples belong in the stable prefix. User messages, retrieved documents, timestamps, session IDs, and other changing fields should stay in the suffix.
What breaks prompt caching in production?
Small differences in the prefix break cache hits. Trailing whitespace, random tool order, drifting JSON key order, daily dates in the system prompt, or user/session IDs inside the prefix can silently disable caching. Deterministic prompt construction matters as much as turning the provider feature on.
When does prompt caching not help?
Prompt caching does not help much for single-shot prompts with no shared prefix, prompts below provider minimum token thresholds, or workloads that burst once and then sit idle for hours. It works best when many requests share a long stable prefix over a short period.