Most teams treating LLM cost as a pricing problem are looking at the wrong variable.

The price per token is set by the provider. You can’t negotiate it. What you can control is how many tokens you send — and that number is almost entirely determined by what you instrument and what you don’t.

Here is the specific observability gap that hides LLM waste, and the exact metrics that expose it.

The standard observability stack misses the point

If you are using a standard APM tool — Datadog, New Relic, Grafana — you probably have:

  • Request count per endpoint
  • Latency percentiles (p50, p95, p99)
  • Error rate
  • Uptime

None of these tell you anything useful about LLM cost. A call that processes 8,000 input tokens and a call that processes 200 input tokens look identical in these metrics. Same request count. Similar latency. Same error rate.

The cost difference is 40x.

The four numbers that actually matter

1. Input tokens per call — this is the cost driver. Output tokens matter too, but input is where waste hides. A bloated prompt or an unnecessarily large context window is invisible in standard metrics and expensive on the bill.

2. Prompt hash — a hash of the input lets you detect duplicate calls. If the same prompt (or near-duplicate) is being sent multiple times, you are paying for the same generation repeatedly. Caching at the prompt level is one of the highest-leverage optimizations available.

3. Result utilization rate — was the generated output actually used? If you are generating three variants and discarding two, you are paying for the discards. If you are generating content for items that never pass a downstream quality gate, you are paying for every rejection.

4. Call origin — which feature, job, or code path triggered this call? Without this, you know you’re burning tokens but not where. Attribution is the difference between a useful metric and a number you can only stare at.

How to add these without rebuilding your stack

You do not need a new observability platform. You need a thin wrapper around your LLM client that logs these four numbers:

async function trackedLLMCall(opts: {
  prompt: string;
  origin: string;  // 'autopilot_variants' | 'aeo_profile' | etc.
  model: string;
}): Promise<{ text: string; usage: TokenUsage }> {
  const promptHash = hash(opts.prompt);
  const start = Date.now();

  const result = await llm.generate(opts.prompt, opts.model);
  const latencyMs = Date.now() - start;

  // Log to your existing sink (Datadog, Axiom, stdout + Loki, whatever)
  log.info('llm_call', {
    origin:        opts.origin,
    model:         opts.model,
    prompt_hash:   promptHash,
    input_tokens:  result.usage.input_tokens,
    output_tokens: result.usage.output_tokens,
    latency_ms:    latencyMs,
    cost_usd:      estimateCost(result.usage, opts.model),
  });

  return result;
}

Five fields. One wrapper function. Every call site just uses trackedLLMCall instead of the raw client.

What the data tells you

Once you have these numbers for a week, sort by cost_usd DESC grouped by origin. The top 3 call origins will account for the majority of your spend. This is where to look.

For each top origin, ask:

  • What is the average input token count? If it’s high, look at what’s in the context. Is all of it necessary?
  • What is the prompt hit rate? If you’re sending the same (or very similar) prompts repeatedly, caching will cut costs immediately.
  • What is the result utilization rate? If generated output is often discarded, gate the generation call on a cheaper upstream filter.

In our case, this analysis revealed that a single background job — autopilot variant generation — was responsible for 70% of token spend. It was generating variants for every RSS item regardless of relevance score, and over half of those items were discarded before the variants were ever used.

Fix: run the relevance scorer first (small model, minimal context, costs cents). Generate variants only if score ≥ threshold. Token spend on that job dropped 65% immediately.

The prompt caching opportunity

Anthropic’s prompt caching is one of the highest-leverage features in the API and one of the least-used. If you have a system prompt or a large context block that stays constant across many calls, caching that block reduces input token cost by 90% for cache hits.

The prerequisite is knowing which prompts are candidates for caching — which requires the prompt_hash metric above. Sort your calls by hash frequency. The prompts that appear most often are your caching candidates.

// Mark the stable prefix for caching
const messages = [
  {
    role: 'system',
    content: [
      {
        type: 'text',
        text: LARGE_STABLE_SYSTEM_PROMPT,
        cache_control: { type: 'ephemeral' },  // cache this
      }
    ]
  },
  { role: 'user', content: dynamicUserPrompt }  // not cached — changes each call
];

If your system prompt is 2,000 tokens and you make 1,000 calls per day, caching saves roughly 1.8 million input tokens per day. At current Sonnet pricing, that’s material.

The summary

LLM cost is an instrumentation problem before it is a pricing problem. You cannot optimize what you cannot see, and standard APM tools are blind to the thing that matters: tokens per call, per origin, per result.

Add the four metrics. Run the query. Find the top origin by cost. Ask whether every call from that origin is necessary.

That one question, answered with data, is where 90% of LLM waste lives.

Frequently Asked Questions

Why don’t standard APM tools like Datadog show LLM cost problems?

Standard APM tools report request count, latency, and error rate, none of which correlate with token consumption. Two LLM calls with identical latency and status codes can differ by 40x in cost if one sends 8,000 input tokens and the other sends 200, making the waste completely invisible without token-level instrumentation.

What is result utilization rate and why does it matter for LLM cost?

Result utilization rate measures what percentage of LLM-generated output is actually used downstream. If a system generates three content variants and discards two, or produces output for items that fail a quality gate, the cost of every discarded generation is pure waste that utilization rate makes visible and actionable.

How does prompt hashing reduce LLM API spend?

Hashing the input prompt before each API call lets you detect when the same or near-identical prompt is being sent multiple times. Identical hashes indicate the response can be served from cache, eliminating redundant API calls and their associated token costs without changing model behavior.

What is the highest-leverage way to cut LLM API costs without switching providers?

Prompt-level caching based on input hashing is one of the highest-leverage optimizations because it eliminates entire API calls rather than reducing tokens at the margin. Combined with call-origin attribution to identify the most wasteful code paths, teams can target reductions without renegotiating provider pricing, which is fixed.