Prompt caching vs semantic caching

Side by side

How prompt caching works at each provider

• OpenAI: automatic for prompts of about 1,024 tokens or more, matched on the beginning of the prompt. Check usage.prompt_tokens_details.cached_tokens. A stable prompt_cache_key helps when many requests share a prefix.
• Anthropic (Claude): opt-in. You mark breakpoints with cache_control. Writing to the cache costs more than normal input; reading from it costs a small fraction. The default lifetime is a few minutes, so it pays off under steady traffic.
• Gemini: implicit caching on recent models, plus explicit context caching for large documents, which also charges for storage time.

The rule that makes all three work: put what never changes first (system prompt, tools, examples) and what changes last (the user's message). One timestamp at the top of the prompt breaks the cache for every call.

messages = [
    {"role": "system", "content": LONG_STABLE_SYSTEM_PROMPT},  # same bytes every call
    *FEW_SHOT_EXAMPLES,                                         # same bytes every call
    {"role": "user", "content": user_question},                 # changes every call: keep it last
]

resp = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
print(resp.usage.prompt_tokens_details.cached_tokens)  # > 0 means the prefix was cached

Provider caches expire within minutes. If your calls are far apart, you pay a cache write and never read it back. And if a router like OpenRouter sends turns of the same conversation to different providers, each provider starts cold.

How semantic caching works, and where it goes wrong

A semantic cache turns each question into an embedding, looks for a stored question that is close enough, and returns that answer. "How do I reset my password?" and "I forgot my password, how do I get back in?" can share one answer, which an exact-match cache would miss.

The risk is a confident wrong answer. "What is the capital of Australia?" and "What is the largest city in Australia?" are close in meaning and have different answers. Nothing errors; the user just gets Sydney instead of Canberra. To stay safe:

• Start strict and loosen only after you read real hit pairs, not just the hit rate.
• Verify near matches before serving them, or skip the cache in the uncertain band.
• Scope the cache per tenant, so one customer never sees another's answer.
• Never cache answers that depend on the user, account data, or the current time.
• Set a lifetime so answers refresh when your docs or product change.

Which one should you use?

• Long fixed prompt, every question different: prompt caching only.
• Same questions in many wordings (support, FAQ, docs): both, semantic first.
• Structured extraction or anything personalized: exact match at most.
• Low traffic with long gaps between calls: neither saves much; route easy prompts to a cheaper model instead.

Using both through Prune

Prune is an AI gateway that runs both layers for you. It checks an exact and normalized cache first, then a semantic cache that verifies near matches before serving them, scoped to your account. On a miss it forwards the call with a stable prefix: it adds Anthropic cache_control breakpoints and an OpenAI prompt_cache_key, so provider prompt caching applies without code changes. Every response says whether it was a hit and what it saved.

# Per request, through Prune (OpenAI SDK pointed at https://api.withprune.com/v1)
extra_headers = {
    "X-Prune-Cache": "semantic",        # off | exact | normalized | semantic
    "X-Prune-Max-Cache-Error": "0.03",  # how strict near matches must be
}
# Response headers tell you what happened:
#   X-Prune-Cache-Hit: true | false
#   X-Prune-Cache-Type: exact | normalized | semantic | miss
#   X-Prune-Cache-Confidence: similarity score on semantic lookups

With a single line of code, Prune cuts your AI bill and protects your team from runaway spend, leaked keys and dangerous calls.

Related

• How to reduce your OpenAI API cost
• Prune vs cache-first gateways
• How Prune works