LangChain OpenAI base URL

1. What you can do inside the framework

LangChain can cache in memory or in Redis. That helps one process, and the cache dies or drifts when you add a second server. Provider prompt caching (OpenAI and Anthropic) is a different switch: it discounts a repeated prefix on one provider. Neither one is a per-customer dollar cap. OpenAI limits spend per organization or project, not per end user of your app.

The prompt caching vs semantic caching page is the difference. The per-user limit page is the cap OpenAI does not give you.

2. Point the client at the gateway

Install langchain-openai. The key in the environment is the Prune key from the dashboard. The OpenAI key is saved under Providers, once.

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o-mini",
    base_url="https://api.withprune.com/v1",
    api_key=os.environ["PRUNE_API_KEY"],
)

# exact: a retry of the same messages can be free. Semantic stays off for agents.
# Customer id is the monthly per-user cap, set in the Prune dashboard.
llm = llm.bind(extra_headers={
    "X-Prune-Cache": "exact",
    "X-Prune-Customer-Id": user_id,
})

For Claude through LangChain, drop the /v1. If the installed package asks for anthropic_api_url instead of base_url, use the same host.

import os
from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(
    model="claude-sonnet-4-6",
    base_url="https://api.withprune.com",  # host only, no /v1
    api_key=os.environ["PRUNE_API_KEY"],
)

3. The same line in the Vercel AI SDK

createOpenAI speaks the OpenAI API, so the base URL includes /v1. Chat and generateText both use this client.

import { createOpenAI } from "@ai-sdk/openai";

export const prune = createOpenAI({
  baseURL: "https://api.withprune.com/v1",
  apiKey: process.env.PRUNE_API_KEY,
});

// model id you would have sent to OpenAI
export const model = prune("gpt-4o-mini");

What you should see

• The first call returns prune_metadata in the JSON body. A second identical call can return X-Prune-Cache-Hit: true.
• With routing on Balanced in Optimize, a short simple prompt may move to a cheaper model. The header X-Prune-Routed-Model names it. On OpenAI-shaped calls, tool calls and JSON schema stay on the model you asked for. A short Claude call can still move. Set routing to Off when the model id must stay put.
• X-Prune-Customer-Id counts that user's spend for the calendar month. Set the dollar budget in the dashboard. Over the cap, the call is 429 with X-Prune-Shield-Code and it does not reach the provider.

The lab number on the homepage is a lab, not this app. Read the receipt after a day of your own traffic.

With a single line of code, Prune cuts your AI bill and protects your team from runaway spend, leaked keys and dangerous calls.

Related

• Limit OpenAI usage per user
• Reduce your OpenAI API cost
• LLM model routing
• Prune docs