How Prune works

Request lifecycle

Your app  →  Prune proxy  →  Shield  →  Cache lookup  →  Optimize  →  Provider  →  Response + prune_metadata

1. Authenticate — every call includes X-Prune-Key (or prune_api_key in the SDK).
2. Shield — rate limits, spend caps, and allowlists run before any upstream call. Rejected requests never reach the model (Spend caps enforced in under 10ms, before the provider call).
3. Optimize — routing, output caps, template hoist, and compression on unique traffic (≥25% median cost reduction on unique (non-cache) traffic in the published benchmark suite).
4. Cache — exact match first. Semantic similarity runs only when enabled via cache headers and the similarity threshold clears. Cache is upside on top of unique-path savings — never the only story.
5. Upstream — OpenAI, Anthropic, Gemini, Bedrock, Ollama, OpenRouter, or one of 6 more direct providers receives the optimized payload.
6. Receipt — prune_metadata with counterfactual $ and unique vs cache path is merged into the response body.

What Shield does on every request

Shield is Prune's fail-closed guardrail layer. If Shield cannot verify your policy (database outage, Redis unavailable), the request is blocked — not passed through unprotected. Controls include:

• Per-key rate limits
• Daily spend caps (atomic Redis counters)
• Model and IP allowlists
• Canary token injection for system-prompt leakage (OWASP LLM07:2025)
• Reversible PII tokenization before upstream calls

Rejections return X-Prune-Shield-Code (e.g. spend_cap, rate_limit) for your application to handle.

Cache layers

Exact cache

Identical request payloads (model, messages, temperature, tools) return a stored answer from Redis. Zero upstream tokens, sub-second latency.

Semantic cache

Similar prompts match via embeddings above your similarity threshold. Optional verified mode runs a judge model before returning a near-match.

Template-aware cache

Requests with stable system templates and variable user slots can hit cache when only the variable portion changes — common in support bots and RAG apps.

Multi-endpoint apps (Focus-style)

Tag calls with X-Prune-Feature-Id (e.g. goal_breakdown, coach) and optional X-Prune-Workload so Customers and receipts group by product surface. Recommended mode (auto) already enables semantic cache — best for chatty one-offs; structured JSON wins on exact cache after warm-up.

Response headers: X-Prune-Pipeline-Mode (wire value) and X-Prune-Pipeline-Label (Recommended / Max savings). First misses with why_no_savings are expected warm-up, not failure.

Latency: overhead vs end-to-end

Published SLOs measure Prune overhead before the provider call (p50 <30ms / p95 <80ms on miss path) — not total generation time. A miss that takes 5–10s is almost always upstream. Cache hits can make the whole call faster than going to the model.

What prune_metadata contains

{
  "cache_hit": true,
  "cache_type": "semantic",
  "tokens_saved": 1840,
  "cost_saved": 0.0042,
  "optimizations_applied": ["template_opt", "schema_compress"],
  "shield_canary_injected": false,
  "route_fallback_used": false
}

Fields vary by request. Shadow mode adds shadow_* fields showing projected savings without changing the upstream model.

How to read a receipt

• cache_hit: true — no LLM call; upstream cost is $0 for that request.
• tokens_saved — input/output tokens avoided vs. an unoptimized baseline.
• cost_saved — estimated dollars kept based on list pricing.
• optimizations_applied — which pipeline stages ran (cache, compression, routing).

Frequently asked questions