Concepts

Caching

Prefix caching is automatic. There is nothing to turn on and no cache-control markup.

When the start of a request matches the start of a recent request to the same backend, the backend reuses its work for the matching tokens. Those are cached tokens.

Billing

Cached tokens are billed at the model's cached_input price instead of its input price. For most models that is about 10% of the input price; see pricing.cached_input in GET /v1/models for each model's exact rate.

cost = (input_tokens − cached_tokens) × input_price
     + cached_tokens × cached_input_price
     + output_tokens × output_price

Prices are per 1M tokens. Batch requests then take 50% off the total.

Seeing cache hits

Where Field
Response header x-utmost-cached-tokens (not on streams)
Chat completions usage.prompt_tokens_details.cached_tokens
Responses usage.input_tokens_details.cached_tokens
Messages usage.cache_read_input_tokens

prompt_tokens and x-utmost-input-tokens include cached tokens. In Messages, input_tokens excludes them, following the Anthropic convention.

We report the cached count the backend returns. If a backend reports none, cached tokens are 0 and you pay the full input price.

Getting more hits

Caches match from the first token. A change early in the prompt makes everything after it miss.

  • Put stable content first: system prompt, tool definitions, reference documents, few-shot examples.
  • Put what changes last: the user's question, retrieved snippets, timestamps.
  • Keep the stable part byte-identical between requests. Do not put a date, request id or random ordering in the system prompt.
  • Keep tool definitions in a fixed order.
  • In a conversation, append turns. Do not rewrite earlier ones.
  • Pin a model for workloads that depend on the cache. Each model, and each deployment of it, has its own cache. With an auto alias, requests can land on different models, and a fallback moves a request to another deployment.

Cache entries expire after a period that depends on the backend and its load. There is no guaranteed lifetime.