What order should LLM cost reductions be applied in?

Prompt caching first, then trim the system prompt, then remove unused tool definitions, then routing, then batch for non-interactive traffic. Fine-tuning comes last, or not at all.

Why is fine-tuning often the wrong cost reduction?

A production team reported fine-tuning for repetitive work as a failure: a retrain whenever the work shifted, quality drifting as the base model was upgraded upstream, and the saving confined to roughly 15% of volume and 2% of spend.

How should fine-tuning cost be treated on the P&L?

As a recurring operating cost, not a one-off project cost. It needs ongoing retraining and evaluation as the base model changes, so booking it as a project understates its run-rate.

How much does prompt caching reduce a bill?

Reported flagship rates put cache reads at about $0.20 per million against $4.00 for fresh input. On a workload with a repeating prefix, total billed tokens typically fall 40 to 70 percent once caching is enabled.

The order of cost reductions

Updated: September 27, 2026

By the LLM CFO team

The consensus answer from teams running LLM cost control in production is not a list of techniques. It is that the order matters more than the list. The same teams report that the optimisation they reached for first — fine-tuning a small model for repetitive work — was the one that failed, on maintenance overhead and quality drift.

The sequence that compounds

Prompt caching. The cheapest engineering hour available. Reported flagship rates put cache reads at $0.20 per million against $4.00 fresh, a 95% discount on the largest line. Anything that repeats across calls — system prompts, tool definitions, retrieved context — is leaving this on the table.
Trim the system prompt. Verbose instructions bill on every call forever. This is an editing task, not an engineering task, which is part of why it gets skipped.
Remove unused tool definitions. Every tool definition is billed on every request whether or not it is called. This is a configuration change with a permanent effect.
Route by task. Only now does routing pay, because a router built on steps 1–3 is already cheaper before it switches a single model.
Batch the non-interactive traffic. A straight discount for anything that is not customer-facing.
Fine-tune last, or not at all.

Why the fine-tuning trap is a finance problem

A fine-tune is not a one-off cost. It is a permanent ops tax: a retrain every time the underlying repetitive work shifts, plus ongoing evaluation as the base model is upgraded upstream. The saving usually lands on 15% of volume and about 2% of spend — a small number wrapped in a large ongoing cost. On a P&L that belongs in run-rate, not in a one-off project line, and a project line understates it.

Why the order is the point

Each step changes the token counts the next step operates on. Trim the prompt before routing and the cheap model carries less on every call. Enable caching before fine-tuning and the fine-tuning budget only has to cover cache misses. Run the sequence in the wrong order and you pay twice: a fine-tune a prompt rewrite would have made unnecessary, or a router that never pays for itself.

Bottom line

Caching, trimming and tool cleanup are cheap, compounding and low risk. Routing and batching come after. Fine-tuning comes last or not at all — because the list is easy and the sequence is where the money is.

← Back to llmcfo.com

The order of cost reductions

The sequence that compounds

Why the fine-tuning trap is a finance problem

Why the order is the point

Bottom line

Related