Inference Cost Reality Check: How to design GenAI features that won’t blow up your unit economics
A GenAI pilot can look cheap. Then you ship it. Usage grows. And suddenly your finance team asks a simple question: what are we paying for, per outcome?
That’s the real shift. In production, you’re not managing “model spend.” You’re managing a product that can create cost spikes through normal user behavior—longer inputs, retries, peak-hour traffic, tool chains, and those “just one more regen” clicks.
So let’s keep it practical. Here’s how to design GenAI features that grow without wrecking unit economics.
First, pick a unit that your business actually cares about
Cost per 1K tokens is useful. But it’s not a unit economics story.
A better unit is cost per outcome. That means:
Cost per outcome = (ALL GenAI costs in a period) ÷ (number of successful outcomes)
“Successful outcomes” should be things your leadership already tracks:
- Cost per resolved ticket (support workflows often land in the $1–$5 per ticket range)
- Cost per processed page/field (document workflows often fall around $0.50–$2 per page)
- Cost per qualified lead (sales workflows can land around $4–$12 per lead)
This framing does two things:
- It makes GenAI spend auditable.
It forces the hard question: are we paying for results, or for “messages sent”?
What you pay for when someone clicks “Generate”
Inference cost is not only tokens. Tokens are usually the biggest line item, but production cost is a bundle:
- Tokens: input + output
- Tool calls: retrieval, reranking, function calls, agent steps
- Retries and regenerations: user retries, timeouts, fallbacks
- Latency overhead: slow responses increase concurrent load and duplicate requests
- Quality overhead: human review, evals, monitoring (often non-trivial)
A common pattern is that token volume contributes about 50–70% of the run cost, while human-in-the-loop and operational overhead can add meaningful weight (think 15–25% for review and 20–30% for infra in many setups).
That’s why “we’ll just switch models later” is not a cost plan. Cost is driven by design.
The three multipliers that make costs jump
1) Tokens: the obvious one, but the real issue is variance
In production RAG Q&A, a typical request often sits around 1,000–3,000 tokens total, depending on how much context you stuff in and how long the answer is. A common breakdown looks like:
- 300–600 tokens for system + user prompt
- 700–1,500 tokens for retrieved context
- 200–500 tokens for the answer
Summarization can be heavier. Inputs can range from 1,000 to 10,000 tokens, with outputs around 200–800 tokens, depending on how strict you keep the summary.
But the real trap is not the average. It’s the tail.
In many production systems, token use is heavy-tailed. Your P95 requests (the top 5% biggest ones) can be 3–8× larger than the median. It’s common to see P95 input tokens at 2,000–8,000 and outputs 500–2,000. Those few outliers can dominate the bill.
2) Latency: it quietly increases cost and reduces trust
Latency is not only a UX problem. It drives cost in two ways:
- User behavior: slow responses trigger retries and regens.
- Systems behavior: slow calls keep more sessions open, raising concurrency load.
Longer context lengths hit latency hard. P95 latency can move from sub-500ms around 2K tokens to 2–10 seconds when you’re in the 8K–32K context zone, depending on model and serving setup.
And TTFT (time to first token) often climbs with input size. A rough production relationship shows TTFT increasing around 0.20–0.24ms per token at P95, which adds up fast once you let prompts grow unchecked.
3) Retries and regens: the budget leak nobody plans for
In chat-like products, “regenerate” behavior in production often sits around 5–15%. Many enterprise teams try to keep it under 8% because regen is basically “pay twice for one answer.”
Retries in tool-heavy or agentic flows can be worse. It’s common to see 8–20% retry rates when tool calls are slow or brittle. Timeouts often contribute 40–60% of those retries, followed by parse/schema issues and rate limits.
Here’s the blunt truth: if your feature needs three attempts to get onee usable result, your unit economics are already off.
The design patterns that keep unit economics sane
Pattern 1: Budget the feature, not just the modl
Every GenAI feature should ship with explicit budgets:
- max input tokens
- max output tokens
- max tool calls
- max retries (hard cap)
When the budget is exceeded, don’t crash. Degrade gracefully:
- ask one clarifying question
- narrow scope (“Summarize section 3 only”)
- switch to a cheaper mode (“short answer”)
Budgeting feels restrictive until you see your P95 spend curve.
Pattern 2: Route traffic like a smart call center
Most requests are simple. Treat them that way.
A “small model first” approach can handle 60–80% of traffic on cheaper models and escalate only 20–40% to premium models. Typical savings reported for routing/cascade setups land around 50–75%, with small quality loss when tuned well.
A practical way to do this:
- start with a triage model that classifies intent + complexity
- only escalate when confidence is low or the task is truly complex
This is not fancy. It’s the same logic as tier-1 vs tier-2 support.
Pattern 3: Cache aggressively (because users repeat themselves)
Caching is one of the few levers that improves cost and latency at the same time.
Three types matter:
- Semantic caching: caches answers for “same meaning, different wording.” Case studies show 40–80% cost reduction with hit rates around 40–69% in the right workflows.
- Prompt caching: reuses repeated prompt prefixes. Reported savings include 45–90% input token reduction and meaningful TTFT gains when hit rates are good.
- Retrieval caching: avoids repeated vector DB calls for similar questions; hit rates can vary widely (40–93%) based on thresholds and workload shape.
If you’re using tools like Redis (for semantic cache), or serving stacks like vLLM/TensorRT-LLM, caching is often the simplest “big win” you can ship early.
Pattern 4: Keep RAG disciplined
RAG becomes expensive when it becomes lazy.
Rules that work:
- retrieve only when needed (don’t fetch context “just because”)
- cap how many chunks you add
- cap total retrieved tokens
- force concise answers unless the user asks for depth
If your RAG feature quietly turns every question into a 10K-token prompt, it’s not a RAG feature anymore. It’s a cost bomb.
Pattern 5: Fix retries with UX, not more prompts
A surprising amount of spend is caused by unclear inputs and unclear UI.
Improved prompt UX patterns can reduce retry rates by 30–60%—things like structured inputs, examples, and clear output formats that reduce parse errors and user confusion.
Sometimes the right move is not a better prompt. It’s a dropdown.
Measure it like a product, not like a science project
If you want predictable unit economics, you need visibility per feature:
Track:
- tokens per request (avg and P95)
- TTFT and total latency split (model vs retrieval vs tools)
- regen rate and retry rate
- tool calls per successful outcome
- cost per outcome trend (weekly)
Teams also use guardrails like per-request P95 token limits (for example, <8K), daily user budgets, and burn-rate alerts that trigger throttling when spend spikes.
This is just FinOps thinking applied to GenAI.
Conclusion
Inference cost blow-ups rarely come from a single bad decision. They come from a feature that ships without budgets, without routing, without caching, and without telemetry that flags the P95 tail before it becomes the average.
GenAI needs to be designed like any other high-traffic system: with constraints, fallbacks, and clear success criteria. That’s how you keep cost tied to outcomes, not to usage noise.
If you’re shipping GenAI features and want a quick unit economics review—tokens, latency, retries, and the design patterns that reduce spend—reach out to iauro.
www.iauro.com or email us at sales@iauro.com
