Prompt Caching

Many requests start the same way: the same system prompt, the same tools, the same long document. Models that support caching keep that start for a few minutes and charge much less to read it again. A cache read often costs a tenth of the input price or less.

How to get cache hits

• Put what stays the same first. System prompt, tools and documents go at the start; the user’s new message goes last. The cache only matches a prompt from its first token.
• Send it again soon. Caches last minutes, not hours.
• Make it long enough. Hosts only cache prompts past a minimum length, often around 1,000 tokens.

Flow AI sends a repeated prompt to the same host where it can, so the cache you built is the one you hit.

Automatic or marked

Some labs cache for you; others need you to mark where the cached part ends.

response = client.chat.completions.create(
    model="<anthropic model id>",
    messages=[
        {
            "role": "system",
            "content": [{
                "type": "text",
                "text": LONG_INSTRUCTIONS,
                "cache_control": {"type": "ephemeral"},
            }],
        },
        {"role": "user", "content": "Summarise section 3."},
    ],
)

Seeing it

usage.prompt_tokens_details.cached_tokens shows how many input tokens were read from the cache. The Logs page shows cached tokens and the cache price for each request.

Billing

Cached input is charged at the model’s cache read price. Models that charge to write a cache list a cache write price too; you pay it on the request that fills the cache. Both are on the model’s page. See Pricing.