The dashboard said everything was fine. Latency: normal. Error rate: zero. Throughput: steady.
The bill said otherwise.
We were burning through LLM API tokens at a rate that made no sense given the actual feature usage. The gap between “what the metrics showed” and “what we were paying for” was a 10x difference. One of those numbers was lying, and it wasn’t the bill.
This is the story of finding that lie, and the fixes that cut token consumption by 90% without removing a single feature.
The instrumentation gap
Most observability setups track the things that are easy to count: HTTP requests, response times, error codes. LLM usage is different. The cost isn’t in the request count — it’s in the token count per request, and those vary by orders of magnitude depending on what you’re passing as context.
Our setup was logging llm_call: 1 for every generation. What it wasn’t logging:
- Input token count per call
- What was actually in the context window
- Whether the context had changed since the last call
- Whether the response was even used
Without those four numbers, the dashboard was technically accurate and completely useless. We had high observability of the wrong thing.
What we found when we actually looked
Once we added token-level instrumentation — input tokens, output tokens, prompt hash, cache hit rate — the picture changed immediately.
Finding 1: We were re-sending full context on every call.
A feature that generated social post variants was passing the entire archive of previous posts as context on every single generation. The idea was to prevent repetition. The reality: 4,000 tokens of context per call, most of it unchanged between calls, zero cache benefit because we weren’t using prompt caching.
Fix: semantic dedup at the retrieval layer. Instead of sending all previous posts, we embedded the new request and pulled the 3 most similar past posts. Context dropped from ~4,000 tokens to ~400. Same repetition prevention. 10x fewer tokens.
Finding 2: We were generating content nobody asked for.
An autopilot job was generating three LinkedIn variants for every RSS item that came in, regardless of relevance score. Most of those items scored below the threshold where we’d ever schedule them. We were paying to generate posts for content we’d never publish.
Fix: gate generation on relevance score. Score first (cheap: small model, minimal context). Generate only if score ≥ threshold. This alone eliminated ~60% of generation calls.
Finding 3: Retries were compounding the problem.
The generation function had a retry wrapper that re-submitted the full request on any failure, including rate limit errors. A rate limit response costs zero tokens — the request was rejected before processing. Retrying immediately just hit the limit again. And again. And again, each time burning the overhead of the HTTPS round-trip with no chance of success.
Fix: exponential backoff with jitter, rate limit responses counted as non-billable, and a circuit breaker that stopped retrying after 3 consecutive rate limits and queued the job for the next minute instead.
The 90% number
Here is where it came from:
| Change | Token reduction |
|---|---|
| Semantic context retrieval | ~75% reduction in input tokens per call |
| Relevance gate before generation | ~60% fewer calls |
| Retry fix (stop hammering rate limits) | ~15% reduction in wasted calls |
These compound. Fewer calls × smaller context per call = the bill you’d expect if someone had thought about this in the first place.
The feature set didn’t change. The quality didn’t drop. What changed was that we stopped generating things we didn’t need, stopped sending context that wasn’t relevant, and stopped retrying things that couldn’t succeed.
What good LLM instrumentation looks like
If you’re running LLM features in production, these are the numbers worth tracking:
Per call:
- Input token count
- Output token count
- Prompt hash (to detect duplicate/near-duplicate calls)
- Whether the result was used (or discarded)
- Whether a cached response was returned
In aggregate:
- Token cost per user action (not per API call)
- Cache hit rate
- Generation calls that produced a result below your quality threshold
- Retry rate by failure type (rate limit vs. error vs. timeout — these have different fixes)
The goal is to make waste visible. You can’t fix what the dashboard hides.
The broader pattern
This isn’t really an LLM story. It’s an instrumentation story. The LLM was doing exactly what we told it to do — which was the problem. We had told it to generate on every item, with full context, and to retry on failure. It complied perfectly.
The mistake was treating “it ran without errors” as evidence that it ran efficiently. Those are different things, and in systems where the unit cost is variable (token pricing, not flat per-request pricing), the difference shows up on the bill before it shows up in the metrics.
Measure the right things. The dashboard will tell you whatever you instrumented it to tell you. Make sure that’s actually what you need to know.
What to do tomorrow
If you’re running LLM features and haven’t done this yet:
- Add token-count logging to every LLM call — input and output, separately
- Log whether each generation result was actually used downstream
- Check your retry logic — are you retrying rate limits with no backoff?
- Find your highest-volume generation path and ask: is every call here necessary?
One afternoon of instrumentation work. The bill will tell you what to fix next.
Frequently Asked Questions
Why don’t standard observability dashboards catch high LLM API costs?
Standard observability tools track request count, latency, and error rates — none of which correlate to LLM token cost. Token cost is driven by the size of the context window per request, which can vary by orders of magnitude. Without logging input token count and prompt content per call, dashboards can show healthy metrics while costs run 10x higher than expected.
What is the most effective way to reduce input token count in LLM API calls?
The highest-impact change is replacing full context re-sends with semantic retrieval. Instead of injecting an entire dataset into every prompt, embed the current request and retrieve only the top 2-5 semantically similar items. This can reduce input tokens by 10x while preserving the functional intent of the context, such as repetition prevention or style matching.
How does prompt caching reduce LLM API costs?
Prompt caching allows LLM providers to skip reprocessing prompt prefixes that haven’t changed between calls. If a large system prompt or static context block is sent repeatedly, caching eliminates the token processing cost for that portion. The prerequisite is that the cached content must be structurally consistent across calls — dynamic or shuffled context blocks will miss the cache.
What metrics should be logged to properly monitor LLM API usage?
At minimum, log input token count, output token count, prompt hash, cache hit rate, and whether the model response was consumed by the application. Request count alone is insufficient. Prompt hash enables detection of redundant context, and tracking response utilization surfaces cases where tokens are spent on generations that are never shown to users.
