Most teams reach for an LLM gateway for the same three reasons: they are tired of every service holding its own provider key, they cannot answer "what did we spend on Claude last month, by team", and a provider outage took production down because there was no fallback.
A hosted gateway solves those. Self-hosting solves them too, and additionally answers a fourth question that a hosted gateway cannot: where does the prompt text physically go? If you are in healthcare, finance, defence, or anywhere with a data residency clause in the contract, that fourth question is the only one that matters.
This is a practical guide to running one yourself. It covers what self-hosting genuinely gets you, how to read the licence before you commit, and a working deployment you can copy.
What self-hosting actually buys you
Be specific about this, because the honest list is shorter than the marketing list.
Data never leaves your boundary. Prompts and completions carry the most sensitive text your company produces: patient notes, deal terms, source code, customer support transcripts. A self-hosted gateway means that text transits your network to the model provider and nowhere else. No third-party control plane sees it, so there is no third party to put in your subprocessor list, your DPA, or your incident response plan.
You control egress. A gateway you run is a chokepoint you own. You can deny-list model providers, force all traffic through a corporate proxy, and prove in an audit that no application can reach api.openai.com directly because the network policy does not allow it.
No per-request markup and no seat pricing. You pay for compute you already run. At high volume this matters; at low volume it does not, and the honest answer is that a hosted gateway is cheaper than your engineer's time.
No vendor availability coupling. Your gateway's uptime is your uptime. That is a benefit only if your operational maturity is higher than the vendor's, which for most teams it is not. Be truthful with yourself here.
What self-hosting does not buy you: less work. You now own upgrades, TLS, secrets, database backups, and the pager. Budget for it.
Read the licence before you commit
"Open source" is doing a lot of work in this category, and the differences are load-bearing. Everything below was checked against the actual LICENSE file in each repository on 31 August 2026.
Fully permissive, no carve-out. Bifrost is Apache-2.0. Envoy AI Gateway is Apache-2.0. Apache APISIX, whose ai-proxy plugin fronts LLM providers, is Apache-2.0. Kong is Apache-2.0 (its AI plugins ship in the open-source gateway, though some AI features are Enterprise-only). The Portkey AI Gateway repository is MIT.
Open core with a commercial carve-out. LiteLLM is the most widely deployed option and its root LICENSE states plainly:
Portions of this software are licensed as follows:
* All content that resides under the "enterprise/" directory of this
repository, if that directory exists, is licensed under the license
defined in "enterprise/LICENSE".
* Content outside of the above mentioned directories or restrictions
above is available under the MIT license as defined below.
The enterprise/LICENSE.md then requires a paid BerriAI Enterprise licence for production use of that directory, "for the correct number of user seats". This is a completely reasonable business model, and LiteLLM's MIT core is genuinely usable on its own. But you should know before you build a compliance story on top of it that some features you may want, such as SSO and certain audit capabilities, live behind that boundary.
The practical rule: git clone the repo, read the root LICENSE, then look for an enterprise/ or ee/ directory and read that licence too. Two minutes of reading avoids a difficult conversation with procurement six months later.
Watch for the product moving. In 2026 this category consolidated. Portkey's hosted product is now Palo Alto Networks' Prisma AIRS AI Gateway, and portkey.ai carries the banner "Portkey is now PRISMA AIRS AI Gateway, generally available for all enterprises" (observed 31 August 2026). The MIT-licensed gateway repository is still public, with its most recent push on 25 May 2026. If you are picking infrastructure with a five-year horizon, check commit recency as carefully as you check the licence.
The deployment
Here is a working stack. Everstack is used as the concrete example because it is what we build, and because the gateway ships in the same binary as the observability that makes it useful. The shape generalises: every gateway in this category needs a metadata store, and every one worth running needs somewhere to put telemetry.
Three dependencies: Postgres for metadata (keys, provider config, routing rules), ClickHouse for telemetry (traces, spans, logs), Redis for caching and rate limit counters.
# docker-compose.yml
services:
postgres:
image: postgres:16
environment:
POSTGRES_USER: everstack
POSTGRES_PASSWORD: changeme
POSTGRES_DB: everstack
volumes:
- pgdata:/var/lib/postgresql/data
clickhouse:
image: clickhouse/clickhouse-server:24.3
volumes:
- chdata:/var/lib/clickhouse
ports:
- "8123:8123"
redis:
image: redis:7-alpine
everstack:
image: ghcr.io/everstacklabs/everstack:latest
ports:
- "8089:8089"
environment:
EVS_DATABASE_URL: "postgres://everstack:changeme@postgres:5432/everstack?sslmode=disable"
EVS_CLICKHOUSE_URL: "clickhouse://clickhouse:9000/everstack"
EVS_REDIS_URL: "redis://redis:6379"
EVS_AUTH_SECRET: "generate-a-real-secret"
depends_on:
- postgres
- clickhouse
- redis
volumes:
pgdata:
chdata:
Bring it up and check health:
docker compose up -d
curl -fsS http://localhost:8089/debug/healthz
Budget 4 GB of RAM minimum and 8 GB if you want ClickHouse to be pleasant. The 20 GB of disk in the prerequisites is for volumes, and telemetry is what fills it. Plan retention on day one, not after the disk alert.
Add a provider and issue a key
Open the dashboard on port 8089. The first account to register becomes the owner, so register immediately rather than leaving that open. Then add a provider under Vault, and create an application API key.
The important detail is that your applications never see the provider key. They get an Everstack key, the gateway holds the OpenAI or Anthropic credential, and rotating the provider key is a config change rather than a redeploy of nine services.
Send a request
The gateway is OpenAI API-compatible, so the migration is a base URL change:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8089/openai/v1",
api_key="YOUR_EVERSTACK_API_KEY",
)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
Anything speaking the OpenAI wire format works unchanged: the official SDKs, LangChain, LlamaIndex, Vercel AI SDK, and most internal HTTP clients. That compatibility is the reason this category standardised on the OpenAI schema, and it is why a gateway migration is usually a day of work rather than a quarter.
Confirm you are actually getting observability
This is the step teams skip, and it is the one that makes a gateway worth running. Open the traces view and confirm your request is there with its model, latency, token counts and cost. If the gateway you chose does not do this by default, you have deployed a proxy, not a gateway, and you will end up bolting a separate tracing stack onto it. Everstack writes OpenTelemetry spans for every request without any SDK in your application, which you can read more about in our observability overview.
Taking it to production
The Compose file above is a development stack. Five things separate it from something you can page on.
Terminate TLS properly. Put the gateway behind your existing ingress. Do not expose 8089 directly.
Get secrets out of the Compose file. EVS_AUTH_SECRET and every provider credential belong in your secret manager, injected at runtime. Provider keys are the highest-value secret in the deployment: they are bearer credentials against a metered API, and a leaked one is a bill.
Run Postgres and ClickHouse as managed services if you can. The gateway is stateless and easy to operate. The databases are not. If your platform team already runs Postgres well, use it. If not, a managed Postgres removes the majority of the operational risk in this deployment.
Set retention deliberately. Full-fidelity spans with prompt and completion text are the most useful data you will have and the fastest-growing table in the system. Decide the retention window before ClickHouse decides it for you.
Configure fallbacks and rate limits on day one, not after the first incident. The reason to have a gateway is that when a provider degrades, the routing layer fails over without a deploy. Configure the fallback chain while you are calm. The same applies to per-key rate limits, which are your protection against one runaway retry loop consuming the month's budget. Our gateway page covers routing, load balancing and caching in more depth, and the full configuration reference is in the documentation.
When you should not self-host
An honest guide has to include this section.
If you are a team of five with no platform engineer, self-hosting is the wrong call. You will spend more on operating three stateful services than a hosted gateway costs, and you will spend it in the currency you have least of. Use a hosted gateway and revisit the decision when either your volume or your compliance obligations change.
If your only requirement is spend visibility and you have no data residency constraint, a hosted tool gets you there this afternoon.
If you need to be running in production this week and you have no Kubernetes or Compose experience in the team, take the hosted option and migrate later. Because the ecosystem standardised on the OpenAI wire format, that migration stays cheap. That is the genuinely good news about this category: choosing a gateway is a reversible decision, so do not over-think it.
Self-hosting earns its keep when the data cannot leave, when the volume makes per-request markup material, or when you want one control plane over routing, sandboxes, evaluations and traces rather than four vendors with four DPAs. If none of those describe you yet, that is a fine answer. Come back when one of them does.