LLM API providers are charging you twice for output tokens

Under agentic inference patterns current APIs charge you twice for output tokens. Once when they are generated, at output price, and again when they are written to the prompt prefix cache, at cache write price, on the subsequent API call.

It doesn’t have to be this way. Open source inference engines, SGLang and vLLM, retain the KV cache for both prompts and generations.

LLM API providers’ inference engines are almost surely capable of this under the hood as well, however current APIs don’t give their users a way to request the retention of KVs for output tokens.

There needs to be a way for API users to signal, or even pay, providers to store output tokens as well.

all APIs only cache the prompt (input), not the output.

LLM APIs allow their users to request that KVs computed for input tokens be retained so that they can be reused on future requests with the same prompt prefix. This is aptly called prompt caching.

This worked fine for chatbots (see appendix), but is no longer sufficient for agents. The core of every agent, the agent loop, always¹ reuses the output from the previous result on the following API request. Because the output from the previous request was not cached we must pay to write it to the cache on the next request.

Let’s walk through a minimal example of what this looks like from the perspective of someone using Anthropic’s Fable 5 through the API. This looks identical for OpenAI and Gemini APIs just with different pricing.

Notice that you pay for assistant tokens with tool calls twice.

For every token generated before the last API request in the agent loop you pay output tokens at least 2 times. First you pay for the tokens as they are generated (result 1). Then on the very next request, after you execute the tools and append the tool results to the context, you pay to add the tokens to the ‘prompt prefix cache’ (request 2). Then you pay to read them from cache on every following request before compaction.

how much are we actually paying for output tokens?

~20-25% more than advertised

Naively, if we are paying output price upon generation and then cache write input price on the next request then we can just add up their costs to get an estimate of how much we are really paying for ‘agentic output tokens’².

output + cache WRITE input = cost of 'agentic output tokens'
$50.00 + $12.50            = $62.50 / 1M tokens

$62.50 / $50.00 = 1.25 => 25% more than advertised output price

output + cache WRITE input = cost of 'agentic output tokens'
$30.00 + $6.25             = $36.25 / 1M tokens

$36.25 / $30.00 = 1.2083 => 20.83% more than advertised output price

note: we have to pay for every token in every request, so ‘agentic output tokens’ will always have some cost for each request. However, as we will see, that cost should be at cache read pricing, not cache write which is 10-12.5x more expensive.

KVs for output tokens don’t have to be computed twice

From the perspective of the prompt prefix cache there is no difference between input and output tokens, it is all just tokens.

Just like they retain the KV cache blocks for input tokens, providers can retain the KV cache blocks created during generation so we no longer have to pay for cache write on the next call.

Let’s take a look at what goes on under the hood with the prompt prefix cache for our two requests in the example above.

request 1:

request 2:

During inference, Key and Value vectors (KVs) are generated for every token. KVs for input tokens are generated all at once during prefill. And the K and V for each output token is generated one at a time during autoregressive decode³.

After decode is complete (because LLM completed a tool call) blocks⁴ of KVs are retained in cache so they can be reused, instead of recomputed, during the subsequent request.

However only KV blocks for input tokens are retained, meaning that KVs for output tokens from request 1 must be recomputed during prefill stage of request 2.

If this sounds redundant that’s because it is. There is nothing preventing them from retaining the KV blocks for the output tokens too. In fact this is exactly what the open source inference engines SGLang⁵ and vLLM⁶ do.

how inference should look within an agent loop

Never prefill tokens you just decoded

Instead of discarding the KVs for output tokens providers can just retain their blocks with the rest of the prompt prefix cache at the end of request 1.

Then on the next request (request 2) all KV blocks from request 1 will be in the prompt prefix cache, saving the inference engine from recomputing anything during prefill.

request 1:

request 2:

Notice how there is no longer any overlap between prefill and decode across requests.

how this looks from the API user perspective

If API providers retain output tokens in the prompt prefix cache then users only have to pay cache read price for every subsequent request that contains them.

Notice how now API users only pay full price for new input or output tokens the first time they appear. From then on they are always ‘cache read input tokens’.

how much money would this save?

~15.9-18.4% on output tokens, depending on the provider.

Now that we understand how prompt prefix caching should work let’s estimate how much this would save API users⁷.

output + cache WRITE input = CURRENT cost of output tokens
$50.00 + $12.50            = $62.50 / 1M tokens

output + cache READ  input = IDEAL   cost of output tokens
$50.00 + $1.00             = $51.00 / 1M tokens

$62.50 - $51.00 = $11.50 extra per 1M output tokens
$51.00 / $62.50 = 0.816 => 18.4% savings

output + cache WRITE input = CURRENT cost of output tokens
$30.00 + $6.25             = $36.25 / 1M tokens

output + cache READ  input = IDEAL   cost of output tokens
$30.00 + $0.50             = $30.50 / 1M tokens

$36.25 - $30.50 = $5.75 extra per 1M output tokens
$30.50 / $36.25 = 0.841 => 15.9% savings

this improves performance too

Time to first token (TTFT):

No⁸ recomputation means less computation done during prefill. and prefill is compute bound so less computation means users get their first token faster.

Cache Hit Rate:

the SGLang Radix Attention Paper defines the “cache hit rate as number of cached prompt tokens / number of prompt tokens”

Since output tokens are part of the subsequent prompt it is trivial to see that having them cached will improve the cache hit rate, up to its theoretical limit.

however it is impossible to reach 100% under this definition since entirely new tokens are appended to the context each model request, e.g. tool results or new user prompts. instead everyone should measure cache read input tokens / total tokens at the end of previous request to have higher signal into true cache reuse and more easily see when a request invalidates the prompt prefix cache.

how can API providers support this?

APIs need to have a way for users to request that the output token blocks are retained in the prompt prefix cache.

For providers with implicit caching like OpenAI and Gemini they can make this change entirely on the backend since they already handle caching for their users.

For providers with explicit caching like Anthropic this requires an update to the API.

proposed Anthropic API implementation

if people are already using automatic caching, Anthropic could extend their top level cache_control object with an optional key.

# call
response = client.messages.create(
    model="claude-fable-5",
    max_tokens=1024,
    ####################################
    cache_control={ # existing automatic caching param
        "cache_output": True, # new, indicates to cache output
        "type": "ephemeral",
        "ttl": "5m"
    },
    ####################################
    system="You are a helpful assistant.",
    messages=[
        {
            "role": "user",
            "content": "do something agentic (call tools in a loop for me) please",
        }
    ],
)

# response
{
  "content": ...
  "usage": {
    "input_tokens": 2048,
    "cache_read_input_tokens": 1800,
    "cache_creation_input_tokens": 248,
    "output_tokens": 503,
    "cache_creation_output_tokens" 503, # new
  }
}

proposed new pricing model

Again, providers where caching is already priced in⁹ don’t need to change anything.

however Anthropic charges extra for storing the KV cache between requests. they charge 25% of the price of input tokens to store these in cache for up to 5 minutes. essentially this is a flat fee paid per token when that token is retained in the prompt prefix cache between API requests. currently it is only applied to input tokens but since there is no fundamental difference between caching input and output tokens it would make sense to have a single price for caching any type of token.

If they continue with the same pricing model they would likely need to add something akin to a cache_retention_per_1M_tokens. let’s assume this would be at the same 0.25x input token price it is currently.

so for Fable 5 this would be $10.00 * 0.25 = $2.50 per 1M tok

output + cache WRITE fee + cache READ input = LIKELY cost of output tokens
$50.00 + $2.50           + $1.00            = $53.50 / 1M tokens

$62.50 - $53.50 = $9.00 saved per 1M output tokens
$53.50 / $62.50 = 0.856 => 14.4% savings

Not quite as good as the 18.4% from above but still a free almost 15% savings.

why don’t APIs already support this?

we don’t yet have a real answer so I will do my best to provide a plausible explanation since the simplest explanation is usually the best one.

APIs were initially built for Chat applications a la ChatGPT

there are 3 primary LLM API formats: OpenAI’s Chat Completions, OpenAI’s Responses, and Anthropic’s Messages. Other providers may have their own format (e.g. Gemini) but they usually have an API that is compatible with either Chat Completions or Messages.

these APIs were designed in the era of chatbot applications like ChatGPT. and when you have APIs with a lot of users it becomes almost impossible to change them. lucky for us, both Chat Completions and Responses use implicit caching so those providers should be able to support this without any changes to the api.

a more speculative reason could be due to lack of incentive. this ‘feature’ would lead to both lower margins¹⁰ and increased memory usage during a shortage.

providers could be in a prisoner’s dilemma situation where the current equilibrium is optimal. until one lab defects by adding this to their API to capture market share, forcing everyone else to follow. Ultimately leaving everyone with lower margins and less memory.

It is practically guaranteed that their internal inference engines¹¹ for RL and internal API use already do this since this is a literal compute multiplier. There is just not enough of an incentive to update the external API since nobody else’s API supports it and this doesn’t impact internal LLM usage.

why did I write this?

my goal with writing this is not to point blame at any lab or provider, nobody is at fault here. I wrote this simply because I want faster and cheaper tokens so that I can get more out of my 5hr and weekly limits.

and apparently you can just tweet and people will fix things.

I also wanted to get some reps in to practice my writing and technical communication.

ty for reading. I hope you learned something about LLM APIs, KV cache retention, and inference.

appendix

what about the final assistant message at the end of a turn?

as we discussed earlier, we expect every assistant message with tool calls to have a follow up API request, with tool results, before the cache expiration.

however eventually there are no more tool calls and the agent loop stops. what should we do with those output¹² tokens? should we cache them too?

if the session gets a follow up request within the next 5 minutes that reuses those tokens even once then it is worth it to cache.

so how do we know if we will get a follow up request?

we don’t … but the application or harness might.

If this session has an active /goal or /loop running then the harness knows it will reuse these tokens immediately.

But if not the application could try to predict¹³ whether or not the user will follow up before the cache expires.

I am sure there will be many more opportunities for similar harness/inference co-design as both continue to evolve.

evolution of LLM API usage

I don’t believe that prompt caching APIs are purposely built to charge you twice for the current most common usage pattern (append-only agentic inference). The issue is that common usage patterns have shifted significantly since the APIs were first designed.

prompt caching was built for chatbots, not agents¹⁴

prompt caching APIs were initially designed for chat applications (e.g ChatGPT) where it allowed users of the API to reuse the cache for the system message across chats for all users.

Chat 1:

Chat 2:

could be the same user or a different user

multi-turn conversations: what if the user asks a follow up question?

prompt caching works alright for this as well. but tbh we could have seen this issue back then. both SGLang and vLLM did.

notice that each request after the first writes the assistant message from the previous turn to the cache.

when should I write to the cache?

if you use the cache even once it is worth the price

cache_write = 1.25 x input cache_read = 0.1 x input

so if you write 1000 tokens to the cache you pay for 1250 tokens then on next API call, with the same prefix, you get a cache hit and read all those 1000 cached tokens at the price of 100 tokens. so you paid for the equivalent of 1350 input tokens.

however if you don’t cache you pay regular price for 1000 tokens then if you make a request with the same prefix you pay regular price again for 1000 tokens. so on the second request you already paid more than if you had cached. not to mention that caching also improves API response time as well.

my takeaway from this is:

if you are going to make another LLM API call with the same prompt prefix, in the next 5 minutes, you should write that prefix to cache

in vanilla¹⁵ multi-turn conversation it is unrealistic to predict whether or not the user will ask a follow up question ex ante so builders on the API either always or never pay the cache write cost depending on their apps usage patterns.

notice how with the shift from vanilla multi-turn conversations to agent loops it actually got easier to predict whether or not the cache will be reused: if the assistant calls a tool in its response then the harness knows it will reuse the cache in the very next request. This is why builders now always pay the cache write cost.

minimal append-only agent loop

messages: list[Message] = [
    SystemMessage(content="You report the weather."),
    UserMessage(content="What is the weather in Phoenix?"),
]

while True:
    assistant_message: AssistantMessage = llm.generate(messages)
    messages.append(assistant_message.content)

    # if there are no tool calls, we are done
    if not assistant_message.tool_calls:
        return assistant_message.content

    # otherwise execute tool calls, append tool results and repeat
    for tool_call in assistant_message.tool_calls:
        tool_result: ToolResultMessage = execute_tool(tool_call)
        messages.append(tool_result.content)