Slashing LLM Costs by 90%: Prompt Caching and Prefix Optimization

1. The Economics of Attention KV Caching

Every LLM inference call recomputes attention over the entire input sequence unless the model provider caches the intermediate key-value tensors from a previous call that shares a prefix. Prompt caching, offered as a distinct billing tier by every major provider, lets you reuse those precomputed KV tensors for the portion of a new request that exactly matches a previously-cached prefix, skipping the compute for that segment entirely and only running the forward pass on the new tokens appended after it.

The cost structure this produces is asymmetric by design: cached input tokens are typically billed at roughly a tenth of the cost of uncached input tokens across major providers, while output token pricing is unaffected by caching, since generation is inherently sequential and cannot be pre-computed. For workloads with a large, stable shared context, a long system prompt, a retrieved document set, or a codebase snippet, prompted repeatedly across many requests, this ten-times cost differential on the input side compounds into the large end-to-end savings the technique is known for.

The mechanism has a latency benefit that is often as valuable as the cost benefit: skipping the forward pass over cached tokens directly reduces time-to-first-token, since the model does not need to run attention computation over content it has already processed, which matters disproportionately for latency-sensitive applications like conversational agents where users notice the delay before the first token more than total generation time.

2. Cache Lifetime, Invalidation, and Provider Differences

A cached prefix is not permanent: providers evict cache entries after a period of inactivity, typically ranging from a few minutes up to roughly an hour depending on the provider and cache tier, and a cache miss silently falls back to full-price, full-latency processing rather than erroring, which means a caching strategy that looks correct in testing can quietly stop delivering savings in production if request patterns become too infrequent or too spread out across time to keep the cache warm.

Providers differ meaningfully in how caching is triggered and billed: some require an explicit cache-control marker in the request identifying exactly which prefix segment should be cached, while others cache automatically based on prefix matching above a minimum token threshold without any explicit opt-in, and cache write costs (writing a new prefix into the cache for the first time) are sometimes billed at a premium over the standard input rate, which needs to be accounted for when estimating savings on the very first request of a session, not just steady-state requests.

Because eviction and billing details are provider-specific and change over time as providers iterate on their caching products, the durable engineering practice is to structure your prompts for cache-friendliness in a provider-agnostic way and treat cache hit rate as a metric you monitor per provider, rather than hard-coding assumptions about exact TTLs or pricing multipliers into your cost model.