Teams that need AI infrastructure inside their own boundary usually assemble it from parts. That works, and it is a reasonable default. But the assembly has a cost that shows up between the components rather than in any of them, and almost nobody budgets for it.
This post lays out the six layers, names the credible open-source option for each with its actual licence terms, and is specific about the integration work in between.
Licences and prices checked against each project's own repository and pricing page in August 2026.
Layer 1: The gateway
One endpoint in front of every provider, with routing, fallback, caching and per-tenant limits. This is the layer that makes everything else tractable, because it is the one place every model call passes through.
Options:
- LiteLLM (57,637 stars): MIT except
enterprise/. Widest provider coverage anywhere. Virtual keys, budgets, RPM and TPM limits, spend tracking and SSO are on the enterprise side. - Bifrost (7,675 stars, Apache-2.0): clean licence, single Go binary, built for low overhead. Guardrails, cluster mode, adaptive load balancing, RBAC and audit logs are Enterprise.
- Kong AI Gateway (44,059 stars, Apache-2.0 core): six free AI plugins, seventeen requiring an enterprise licence including semantic caching, advanced rate limiting and every guardrail. Strong choice if Kong is already in your path.
- Helicone (6,116 stars, Apache-2.0): clean licence, gateway plus observability, supported on-prem on Enterprise. In maintenance mode since Mintlify's acquisition, announced 3 March 2026, which both companies define as security updates, bug fixes and new models continuing rather than new feature work.
The pattern: governance and high availability are what get gated. They are also what you need in month six, which is why gating them works.
Layer 2: Execution
If agents write code, something runs it, and it must not be a bare container. See our longer piece on sandboxing for the threat model.
Options:
- E2B (13,599 stars, Apache-2.0): Firecracker microVMs, excellent DX. The commercial product is hosted; self-hosting is documented in a root-level
self-host.mdand deployed with Terraform, GCP GA and AWS beta. Concurrency is tiered at 20 and 100. - Daytona: fast, well designed, and its open source repository states it has received no updates since June 2026, when core development moved to a private codebase. The
LICENSEfile was deleted the same month, so GitHub reports no licence; the last open release, v0.190.0, is AGPL-3.0. Isolation is contested: the docs say Linux containers by default, the README claims a dedicated kernel. Their hosted product is unaffected. Do not self-host the OSS build as a security boundary. - Modal: hosted only, gVisor isolation, but the answer if you need GPUs, which none of the self-hosted options provide. Budget off its Sandbox rate, roughly three times its standard compute rate.
This layer has the thinnest genuinely-self-hostable-and-maintained set, and it is the one where getting it wrong has the worst consequences.
Layer 3: Memory and retrieval
Options: pgvector if you already run Postgres and want one less system, Qdrant or Weaviate if you need scale and filtering that Postgres struggles with. All are self-hostable with clean licences.
Practical advice: start with pgvector. Most teams do not have a vector scale problem, they have a chunking and retrieval quality problem, and moving to a dedicated store does not fix that.
Layer 4: Observability
Options:
- OpenLIT (2,731 stars, Apache-2.0): one datastore, no paid tier, maintains the
gen_aisemantic conventions with the OTel community. The lightest to operate. - Laminar (3,212 stars, Apache-2.0): cloud and self-hosted have feature parity, three datastores.
- Opik (21,697 stars, Apache-2.0): biggest community with a clean licence, self-hosted loses only user management, five datastores.
- Langfuse (33,900+ stars): MIT core with an
ee/carve-out. Audit logs, retention policies and server-side data masking need a licence key when self-hosting. - OpenLLMetry: instrumentation only, exports to whatever backend you already run. Often the right answer if you already operate observability.
Emit OpenTelemetry regardless of what you pick. It is the only thing that makes this layer replaceable later, and this category has seen enough acquisitions to make that worth insisting on.
Layer 5: Evaluations
Options: Braintrust (best product, self-hosting is Enterprise only), Langfuse (bundled with tracing, some gating), Opik (clean licence, complete self-hosted build), OpenLIT (lighter capability, no paywall).
The layer most likely to be abandoned after two weeks. The reason is almost always that evaluating required a push step nobody maintained.
Layer 6: The agent runtime
Sessions, tools, approvals, retries, durable state. Most teams start with a framework (LangGraph, CrewAI, the ADK) and discover that a framework is a library, not infrastructure: it does not persist sessions across restarts, enforce per-tenant limits, isolate execution or emit operable traces.
This is the layer people most often end up building themselves, and the one that quietly consumes the most engineering time.
The part nobody budgets for
Six systems is not six problems. It is six problems plus the joins between them, and the joins are where the time goes.
Correlation. A trace ID has to survive from the gateway, through the agent runtime, into the sandbox, and onto the evaluation score. If it does not, debugging an agent that wrote bad code means manually lining up timestamps across four dashboards. This is the single most common failure of assembled stacks and it is not visible until you are debugging an incident.
Identity and tenancy. Every layer needs to agree on what a tenant is. If the gateway says tenant_id, the runtime says org, and the sandbox says nothing at all, per-tenant cost attribution is not possible without a reconciliation job you now maintain.
Credentials. Provider keys should live in one place. If the gateway holds them and the sandbox also needs to call a model, the sandbox needs a scoped token, not a copy of the provider key. Getting this wrong is how a prompt injection turns into a bill.
Upgrades. Six projects on six release cadences, each with its own breaking changes. This is a standing tax on someone's week, forever.
The licence audit. Each project has its own line between free and paid, and those lines move. Two of the projects above changed their packaging in 2026. Whoever owns this stack owns re-reading licence files when a project you depend on gets acquired.
When to assemble and when not to
Assemble when you already operate most of these layers, you have a platform team, or you need best-in-class in one specific layer and the others are secondary. A team already running ClickHouse, Postgres and Kubernetes is not adding much by taking on one more service.
Do not assemble when you have two or three engineers total and AI infrastructure is not your product. Six systems is a full-time job for someone, and it will be the person you least want doing it.
The unified option
Everstack is the other approach: gateway, agent runtime, sandboxes, memory, workflows, evaluations and OpenTelemetry-native observability as one Apache-2.0 control plane you self-host, with no separately licensed enterprise directory.
The honest trade: you get correlation, shared tenancy and one upgrade path for free, because they are the same system. You give up best-in-class in each layer. Braintrust's evaluation product is better than ours. LiteLLM's provider coverage is wider. E2B's sandbox DX is more polished. Langfuse has a far larger community than any part of what we do.
That is a real trade and not everyone should make it. The case for a unified control plane is strongest when you have several of these problems at once and a small team, and weakest when you have one problem and a platform team who can do it better.
Where to start
Whichever route you take, in this order:
- The gateway first. It is the layer everything else correlates through, and it is the cheapest to adopt.
- Observability second, emitting OpenTelemetry so it stays replaceable.
- Execution third, if agents write code, and do not compromise on the isolation primitive.
- Evaluations fourth, once you have traffic worth scoring.
- The runtime last, because by then you will know which parts you actually need.
The mistake is starting with the runtime because it is the most interesting layer. It is also the one whose requirements you understand least on day one.