How to reduce your OpenAI API cost

1. Measure where the money goes

You can't cut what you can't see. The OpenAI usage page shows totals per model, not per feature. Every response already includes a usage object, so log it with a tag for the feature that made the call.

resp = client.chat.completions.create(model="gpt-4o", messages=messages)

u = resp.usage
log.info(
    "llm_call",
    feature="support_reply",   # tag every call with the feature that made it
    model=resp.model,
    prompt_tokens=u.prompt_tokens,
    cached_tokens=u.prompt_tokens_details.cached_tokens,
    completion_tokens=u.completion_tokens,
)

After a few days you will usually find one or two features driving most of the bill. Fix those first. Remember that output tokens cost several times more than input tokens on most models, so a chatty endpoint can cost more than a long prompt.

2. Use the cheapest model that passes your evals

Small models are often more than 10x cheaper per token than flagship ones (check OpenAI's pricing page for current rates). Classification, extraction, routing, and short rewrites rarely need the strongest model. Build a small eval set per task, run it against the cheaper model, and only keep the expensive model where quality really drops.

def pick_model(task: str) -> str:
    # Cheap model for classification, extraction, and short rewrites.
    if task in {"classify", "extract", "summarize_short"}:
        return "gpt-4o-mini"
    # Strong model only where your evals show it is needed.
    return "gpt-4o"

3. Cap output tokens

Without a ceiling, a model can ramble and you pay for every token. Set max_completion_tokens per endpoint, ask for concise answers in the system prompt, and use structured outputs when you only need a few fields. Be careful with JSON: a cap that cuts a JSON object in half breaks your parser, so size it to the largest valid response.

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    max_completion_tokens=300,  # a hard ceiling, not a target
)

4. Let OpenAI's prompt caching work for you

OpenAI automatically discounts input tokens it has seen recently, for prompts of 1,024 tokens or more. It only matches on the beginning of the prompt, so order matters: put the system prompt, tool definitions, and few-shot examples first, and the user-specific content last. Watch usage.prompt_tokens_details.cached_tokens to confirm it is working. If many requests share a prefix, pass a stable prompt_cache_key to improve hit rates.

5. Cache whole answers to repeat questions

Prompt caching discounts input. A response cache skips the call entirely. If users ask the same thing (support bots, docs search, FAQ-style agents), store the answer keyed on a hash of the request and serve it on an exact match. A semantic cache goes further and matches questions that mean the same thing using embeddings. Keep it scoped per tenant, set a TTL, and never cache responses that depend on private or time-sensitive data.

6. Trim prompt and history bloat

• Cut system prompts that grew by copy-paste. Every token is paid on every call.
• Send the last few conversation turns plus a short summary, not the full thread.
• In RAG, retrieve fewer, better chunks instead of stuffing the context window.
• Strip HTML, whitespace, and boilerplate from documents before sending them.

7. Move non-urgent work to the Batch API

Evals, backfills, nightly summaries, and bulk classification rarely need an answer in seconds. OpenAI's Batch API takes a file of requests, returns results within 24 hours, and costs about half the normal price.

8. Stop runaway spend before it happens

The most expensive bills often come from bugs and leaks, not normal traffic: an agent stuck in a retry loop, or an API key committed to a public repo. Set per-key spend limits, alert on unusual spikes, detect repeated identical calls, and keep provider keys out of client code and git history.

Doing all of this with one base URL

Each fix above is a project when you build it yourself. Prune is an AI gateway that does most of them for you: it routes easy requests to cheaper models, caps output, caches exact and semantic repeats, blocks runaway spend before the provider call, and keeps your OpenAI key in an encrypted vault. Every response carries a signed receipt showing what the call cost with and without Prune.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.withprune.com/v1",
    api_key=os.environ["PRUNE_API_KEY"],  # your OpenAI key stays in the Prune vault
)

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.model_dump().get("prune_metadata"))  # cost with and without Prune, signed

What it saves depends on your traffic, so we publish one measured number: In our 26 Sep lab, new questions cost about 94% less than sending everything to gpt-4o, and repeats were free. Lab result, not a customer bill. The first unique call is full price, and your own receipts show the real number for every call.

Related

• Prune quick start
• How Prune works
• Prune Check: paste a prompt, see cost and security flags
• Prune vs LiteLLM