LLM cost optimization for CFOs.
A finance guide · July 26, 2026
Cost optimization is not about saying no to AI. It is about removing waste so the best use cases get more investment. CFOs can drive this by asking the right questions and measuring the right outcomes.
1. Focus on the top five endpoints
Almost every AI bill follows a Pareto distribution: a small number of endpoints drive most of the cost. Ask engineering to identify the top five and ignore everything else for the first six weeks. This avoids optimizing the long tail.
2. Demand A/B tests
Every optimization — a cheaper model, a caching strategy, a routing rule — must be A/B tested against a stable quality baseline. Savings that come with quality regression are not savings; they are future churn or rework.
3. Know the levers
- Prompt caching — 50–90% off cached tokens.
- Model routing — 30–50% reduction by matching query difficulty to model capability.
- Prompt compression — 10–25% reduction by removing redundancy.
- Batch routing — 50% off non-urgent workloads.
4. Measure savings correctly
Savings = locked baseline − reconciled actuals, normalized for traffic growth. Anything measured against gateway logs or internal dashboards is approximate. The invoice is the only truth.
5. Make it a habit
Cost optimization should be a monthly operating rhythm, not a quarterly project. Review the top drivers, pick one or two candidates, test, ship, and reconcile.