Open-weight model total cost of ownership.
10 July 2026
Llama 8B is free. Running Llama 8B on an H100 is not. The calculation looks simple—an H100 rents for ~$2 / hour on Lambda Labs or RunPod, Llama 8B serves ~500 requests/second, so your cost per request is $2 / 1800 = $0.0011. But the real math is messier. You pay for idle time when traffic dips. You pay for quantization, optimizations, and eval runs that don't generate revenue. You pay for tail latency and retry loops. Self-hosting an open-weight model is cheaper than a frontier API only above a specific utilization threshold; below it, you're paying rent on unused GPUs.
The break-even calculation
Start with your hourly GPU cost and your expected throughput. On Lambda Labs, an H100 costs $2 / hour. Llama 8B with vLLM reaches 200–400 req/s, depending on batch size and token length. Let's assume 300 req/s as a realistic middle ground, averaging 100 tokens per request, so 30k tokens / sec.
At that throughput, your per-1M-token cost is:
- GPU cost: $2 / hour ÷ 3600 sec = $0.00056 / sec
- 30k tokens / sec means $0.00056 / 30k = $0.0000187 / token, or $18.70 / 1M tokens
Claude 3.5 Sonnet on Bedrock costs $3.00 / 1M input tokens, $15.00 / 1M output. If your workload is mostly input (e.g., classification, retrieval), you're paying $3.00 / 1M and your self-hosted cost is $18.70. Bedrock wins by 6x. If your workload is mostly output (e.g., generation), you're paying $15.00 / 1M and your self-hosted cost is $18.70. They're comparable, but you haven't included ops yet.
Hidden costs of self-hosting
| Cost category | Typical range | Impact |
|---|---|---|
| Idle GPU time (traffic variance) | 20–40% of GPU cost | If you average 300 req/s but peak at 100 and trough at 50, you over-provision and pay for empty capacity. |
| Model quantization and tuning | $2k–$10k one-time | Amortize over expected lifetime; adds setup delay. |
| Eval and monitoring | 10–15% of GPU cost | Running test suites, LLM-as-judge scoring, latency tracking. |
| DevOps and incident response | 0.5–1.0 FTE per cluster | Assumes <5 H100s. Larger clusters need more; smaller might share ops with other workloads. |
| Egress and data transfer | $0.10 per GB (cloud to cloud) | Pulling training data, syncing checkpoints, logs. Often overlooked. |
| Quantization decode overhead | 2–5% latency tax | 4-bit quantization saves GPU memory but adds decode latency. Some requests may timeout. |
Real-world break-even points
The math changes based on model size and utilization:
- Llama 8B on 1 H100: Break-even vs Claude Bedrock is ~500k–1M requests / month (at 100 tokens/req). Below that, use the API. Above that, self-host wins if you can maintain 70%+ utilization.
- Llama 70B on 4 H100s ($8/hr): Break-even is ~4M–6M requests / month. The unit economics are worse (larger model = lower throughput per GPU) unless your workload requires 70B's quality.
- Mixtral 8x7B on 2 H100s ($4/hr): Middle ground. ~2M–3M requests / month to break even. Better throughput-to-quality ratio than pure Llama.
Note: these assume 100 average tokens per request. High-volume, low-context workloads (classification, tagging, retrieval) push the break-even point much higher. Long-context workloads (summarization, reasoning) favor self-hosting earlier.
When to self-host
- You have >5M requests / month of stable, predictable traffic. Enough scale to justify ops overhead and smooth out traffic dips.
- You need model-specific customization (fine-tuning, system prompts, safety tuning). APIs are generic. If you need "our 8B is trained on our docs," self-hosting is the only option.
- Data residency or compliance forces it. Can't send data to Bedrock or Azure because of data sovereignty rules.
- You have spare GPU capacity already (e.g., training cluster during off-hours). Marginal cost is low.
- Frontier models are too expensive for your use case but you need better quality than 8B. Example: You're spending $50k / month on GPT-4 for a task that Claude 3.5 Sonnet can do at 1/5 the cost, but Claude isn't available in your region. Self-host Llama 70B, validate quality, migrate, and save $40k/month.
When NOT to self-host
- Traffic is bursty, spiky, or seasonal. You can't keep GPUs busy.
- You don't have ops expertise in-house. Hiring or outsourcing the headcount is expensive.
- You need multiple models (classification, retrieval, generation) running simultaneously. Multi-model serving on GPUs is hard.
- Latency is critical and your cloud infra is far from users. Inference latency matters for real-time UX ; hosting remote on a cheapskate cloud provider will feel slow.
- You're evaluating a new feature and aren't sure if it'll stick around. Temporary workloads should always use APIs.
The hybrid approach
Most scaled teams don't pick one or the other. They run low-cost, high-volume workloads on self-hosted Llama (e.g., 1M+ requests/month of classification or retrieval), route high-quality or bursty workloads to Bedrock / Azure OpenAI, and use fine-tuned models on APIs (e.g., Azure OpenAI fine-tuned GPT-4) for ultra-sensitive tasks. This requires a routing gateway (LiteLLM, custom logic) that picks the right model based on cost targets, quality requirements, and current load. The operational overhead is real, but the cost efficiency is usually 30–50% better than staying on a single provider.