Inference vs training: where the GPU bill actually goes.
10 July 2026
For most of 2024, teams asked "Should we fine-tune or use an API?" The honest answer was always "use the API unless you've proven fine-tuning pays for itself." In 2025, the economics flipped. Industry surveys now report that ~55–80% of enterprise AI GPU spend flows to inference, not training. That shift changes the game entirely. The lever is no longer "how we train" — it's "how we serve."
Why the ratio flipped
- Training happened once. A five-month engagement to fine-tune a 70B model on your proprietary corpus was a one-time cost spike. Quarterly retrains added a fraction of that back.
- Inference runs forever. Every chat, every agent step, every classification call hits the model once per execution. If you serve 10M daily requests at 2B tokens per request, you're computing 20T tokens / day, every single day. At 2026 pricing, even at `$0.01 / 1M output tokens`, that's $200 / day just to run.
- The frontier models got cheaper. OpenAI, Anthropic, and others dropped list prices 30–50% per year. Many teams stopped fighting API costs and started renting instead. That cut training budgets but didn't touch inference demand.
- Agents and reasoning tokens scaled faster than optimization. Task-level AI adoption exploded (intake, classification, summary, generation). Reasoning models like `o1` and `o3` added complexity that eats compute. The denominator (request volume) grew faster than the numerator (tokens-per-request) fell.
What the spend breakdown actually looks like
Reported ranges by team size and deployment model:
| Scenario | Inference % | Training % | Notes |
|---|---|---|---|
| API-first SaaS (no self-host) | 95–100% | 0–5% | Fine-tuning via API only; rare adoption. |
| Hybrid (API + self-hosted open-weight) | 60–75% | 25–40% | Serving open models for cost-sensitive paths; occasional retrains on new data. |
| Heavy self-host (Llama on GPU cluster) | 55–70% | 30–45% | Monthly / quarterly retrains; daily inference volume still dominates. |
| Research / benchmark-heavy teams | 30–45% | 55–70% | Continuous ablation, architecture search, evals. Rare in production. |
Why inference is the real cost lever now
Batching. Training batches are huge and run once. Inference batches are small, distributed, and continuous. A 10% improvement to batch throughput (via quantization, pipelining, or speculative decoding) runs 24/7 across your entire workload. It compounds. Training improvements help only when you retrain.
Utilization vs idle. A GPU training a model runs hot. A GPU serving inference sits idle between requests unless you aggregate load or use continuous-batch mode. Most inference deployments waste 60–80% of GPU capacity waiting. Training rarely does.
Model size, not accuracy. For inference, smaller models (Llama 8B vs 70B, `text-embedding-3-small` vs `text-embedding-3-large`) have linear cost benefits. For training, they have sub-linear accuracy losses. Serving 100 requests on an 8B model costs less than 100 on a 70B, no tradeoffs. The incentive to compress is relentless.
Routing and triage. You can't skip training on hard examples, but you can route cheap answers to a weaker model (classification, summaries, retrieval) and send only ambiguous cases to a frontier model. That scales inference efficiency across your entire stack. Training doesn't have an equivalent lever.
The implications for cost control
- Invest in serving infrastructure, not model training. Your money is in request volume, token volumes, and model parallelism. Deploy a vLLM cluster, not a FSDP training script.
- Quantization and pruning matter now. A 2-bit quantized Llama 70B fits on one H100 and runs 4x faster than the base model. Training impact is unproven. Serving impact is immediate and measurable.
- API routing is not a compromise. Bedrock, Azure OpenAI, and Vertex are often cheaper than self-hosting below a throughput threshold. The old "API vs self-host" binary is obsolete. Multi-cloud serving with dynamic routing is now table stakes for cost-serious teams.
- Evals are now infrastructure cost. If 80% of your bill is inference, 20% of that is probably evaluation (LLM-as-judge, synthetic data generation, test coverage). Instrument evals as carefully as you instrument production serving.
Aligning incentives across engineering and finance
Training spend is easy to measure and blame: "We spent $500k on a fine-tuning run that performed 2% better." Inference spend is invisible: "We didn't optimize the batch size and lost $2M to idle GPU."
The fix is tagging. Every request needs metadata: model size, batch size, latency, user ID, cost tag. Aggregate by model, feature, and team. Report weekly. Inference cost should be as visible as training spend. Without that visibility, engineering optimizes for latency, finance asks "why is the bill up 40%?", and no one changes behavior.