Layer 1: The gateway
One endpoint in front of every provider, with routing, fallback, caching and per-tenant limits. This is the layer that makes everything else tractable, because it is the one place every model call passes through.
Open-source AI infrastructure
How to Build a Fully Self-Hosted AI Stack
The six layers of a production AI stack, which open-source project to use for each, where the licence lines fall, and the integration work between layers that nobod…
Layer 2: Execution
If agents write code, something runs it, and it must not be a bare container. See our longer piece on sandboxing for the threat model.
Layer 3: Memory and retrieval
Options: pgvector if you already run Postgres and want one less system, Qdrant or Weaviate if you need scale and filtering that Postgres struggles with. All are self-hostable with clean licences.
Layer 4: Observability
OpenLIT (2,731 stars, Apache-2.0): one datastore, no paid tier, maintains the genai semantic conventions with the OTel community. The lightest to operate. Laminar (3,212 stars, Apache-2.0): cloud and self-hosted have feature parity, three datastores. Opik (21,697 stars, Apache-2.0): biggest community with a clean licence, self-hosted loses only user management, five datastores. Langfuse (33,900+ stars): MIT core wit…
Layer 5: Evaluations
Options: Braintrust (best product, self-hosting is Enterprise only), Langfuse (bundled with tracing, some gating), Opik (clean licence, complete self-hosted build), OpenLIT (lighter capability, no paywall).
Layer 6: The agent runtime
Sessions, tools, approvals, retries, durable state. Most teams start with a framework (LangGraph, CrewAI, the ADK) and discover that a framework is a library, not infrastructure: it does not persist sessions across restarts, enforce per-tenant limits, isolate execution or emit operable traces.