The LLM Cost Engineering Playbook
How we cut a customer's monthly LLM bill 84% without touching accuracy — routing, distillation, caching and the boring engineering behind every dollar.
The $180K/month wake-up call
A SaaS customer scaled a GPT-4 feature from beta to their full base in six weeks. The bill went from $6K to $180K/month. The feature was popular, the finance team was livid, and the engineering leader was asked whether AI was 'even worth it'. We rebuilt the pipeline in three weeks. The bill landed at $28K/month at higher accuracy — an 84% cut. Nothing in the playbook is clever. Everything in it is boring.
Route before you optimise
The single highest-leverage change is a routing layer in front of your models. A cheap classifier decides: does this query need the frontier model, the mid-tier model, or a fine-tuned small model? On the SaaS customer, 71% of queries were confidently answerable by GPT-4o-mini after fine-tuning. Routing alone cut cost 58% before any other optimisation.
Distillation is underrated
For any workflow with enough traffic, distilling a frontier model's outputs into a fine-tuned small model is a 5–10x cost reduction with 1–2% accuracy loss. We ship distillation pipelines that continuously collect frontier-model outputs on a sampled percentage of live traffic, retrain nightly, and A/B into production weekly. Boring, effective.
Prompt caching is free money
System prompts and shared context should be cached at the provider level. OpenAI, Anthropic and Bedrock all price cached tokens at a fraction of fresh ones. Restructuring prompts so the stable parts come first — with a per-user delta at the end — routinely cuts input-token cost 40–70% for chat surfaces.
Streaming, batching and structured outputs
Streaming responses lowers perceived latency, not cost — but pairing it with early-exit logic on classifier-style tasks can. Batching non-user-facing workloads unlocks 50%+ discounts on OpenAI Batch and Bedrock. Structured outputs (JSON schema) reduce reprompt loops, which are the invisible cost sink of hasty deployments.
Kill the retry storms
A poorly-configured retry policy is a licence for your model bill to explode during any provider hiccup. Exponential backoff with jitter, per-tenant rate limits, and circuit breakers on tool calls are boring reliability engineering that pays for itself the first time a provider has a bad afternoon.
Measure per-outcome, not per-token
The right unit is cost per successful task, not cost per token. When you measure per-token you optimise for shorter outputs, which often means worse outputs, which means more retries, which means more tokens. When you measure per successful outcome, the right optimisations become obvious — often 'use the bigger model less often' rather than 'use the smaller model everywhere.'
Key takeaways
- Route by task complexity before optimising anything else
- Distil frontier outputs into fine-tuned small models continuously
- Restructure prompts to maximise provider-level prompt caching
- Measure cost per successful task, not cost per token
T7 Research
Enterprise AI Research Group
T7 Research is the research arm of T7 Solution, focused on benchmarking LLMs, evaluating RAG patterns, and compiling implementation playbooks for enterprise technology leaders.