Context compaction is a cost lever.
10 July 2026
A customer support chatbot that remembers 50 turns of conversation with a user pays for all 50 turns as input tokens on every subsequent request. An autonomous agent running for 10 steps accumulates a full transcript that it reads before taking step 11. Both cases can reduce input tokens 50–70% by trimming or compressing the context window—dropping old turns, summarizing history, or replacing raw logs with structured memory. The cost savings are real, but so are the failure modes. Context compaction is a lever, not a free lunch.
Where context compaction wins
- Long-running customer support chats. After 30 turns, most users are asking follow-ups on the same problem or problem variants. Summarizing the first 20 turns into a 3-sentence executive summary cuts context 70% with minimal signal loss.
- Multi-turn agents. A procurement agent that gathers requirements, checks inventory, and books delivery can distill each step's reasoning into 2–3 key facts for the next step, then discard the full log. Reduces per-step cost as the agent runs longer.
- Retrieval-based workflows. Instead of passing raw search results + full chat, extract salient facts, deduplicate, rank, and pass only the top 3 results. Cuts context while preserving signal.
- Bulk processing with repeated context. A batch job that analyzes 1000 documents with the same instructions can embed the instructions once (prompt caching) and reference it 1000 times, rather than re-encoding them each time.
Where it breaks
- Fact-dependent workflows. "Have I already told you I'm diabetic?" An agent that forgets a critical detail buried in turn 3 may contradict it in turn 12. Medical, financial, and legal uses are high-risk. You need retrieval, not trimming.
- Dependency chains. A conversation where later turns directly reference details from turn 2. Summarization breaks the chain. Example: "As I mentioned earlier, I need X. Can you also deliver Y?" If the summary doesn't include "I need X," the model has no context for "also".
- Reasoning transparency. For audit and debugging, you need the full trace. A customer disputes a recommendation; you pull the chat log to understand the reasoning. A trimmed log is useless for this.
- Rare-but-important context. "I'm allergic to peanuts." Buried in turn 5. If your summary strategy is "keep only the last 5 turns," you lose it. Structured memory helps here, but adds infrastructure.
Compaction strategies and their tradeoffs
| Strategy | Cost savings | Risk |
|---|---|---|
| Drop turns older than N steps | 20–40% | Loses history ; good for simple chats, bad for reasoning. |
| Summarize old turns into 1–2 sentences | 40–60% | Lossy ; depends on summarization quality. |
| Extract facts into structured memory (JSON) | 50–70% | Requires schema design ; hallucination risk if memory is incomplete. |
| Retrieval instead of full context | 60–80% | High latency ; misses context if retrieval query is weak. |
| Prompt caching (OpenAI / Anthropic) | 90% on cached tokens | Only helps if context repeats ; cache invalidation is a new cost. |
How to measure the tradeoff
Offline eval. Take 100 real conversations, apply your compaction strategy, and re-run the model with the compacted context. Have an LLM-as-judge score whether the response quality changed (no change, degraded, or better). Sample 20 for human review. If <5% show degradation, the strategy is safe to deploy.
Production canary. Route 5% of traffic to the compacted path. Track latency, error rate, and user satisfaction metrics (thumbs-up/down, conversation completion rate). If no difference over 1 week, expand to 50%. If worse, rollback.
Cost accounting. Track input tokens, output tokens, and cost separately for both paths. A 50% reduction in input tokens might save $100 / day but cost $5 / day in extra summarization calls. The net is still positive, but the real savings is lower than the token count suggests.
Combining with prompt caching
Prompt caching (OpenAI `$0.90 / 1M` for cached input vs `$3.00 / 1M` for regular input; Anthropic ~90% discount on cache-read tokens) is context compaction for the static parts. If your agent always starts with the same system prompt + 50 KB of operational guidelines, cache those. Then use live context compaction for the conversation history that's unique to each request.
The combination is powerful: cache the steady-state context (system prompt, retrieval docs, business rules) at 90% discount, then trim the conversation history aggressively. You end up with 80–95% savings on input tokens for multi-turn flows.
Common implementation mistakes
- Forgetting to measure baseline quality. You optimized for cost and now you don't know if the 30% quality drop you're seeing was already there. Always measure before and after.
- Over-trimming on the first deploy. Start with a small compaction (drop turns >100 steps old). Measure. Then get more aggressive. You can't undo silent failures.
- Assuming all turns are equal. Dropping the most recent turn is worse than dropping turn 1. Recency matters. Structure your trimming to preserve recent context.
- Not including compaction cost in the savings calculation. Summarization calls cost tokens too. If you save 1000 input tokens by summarizing but spend 100 tokens on the summarization itself, your net savings is 900 tokens, not 1000.