inference-gateway
An open-source, cloud-native, high-performance gateway unifying multiple LLM providers, from local solutions like Ollama to major cloud providers such as OpenAI
What is inference-gateway?
An open-source, cloud-native, high-performance gateway unifying multiple LLM providers, from local solutions like Ollama to major cloud providers such as OpenAI, Groq, Cohere, Anthropic, Cloudflare and DeepSeek.
Overview of inference-gateway
What it is
Inference Gateway is an open-source, cloud-native, high-performance proxy server that unifies multiple LLM providers behind a single OpenAI-compatible API. It simplifies interacting with various language models, including local solutions like Ollama and major cloud providers such as OpenAI, Groq, Cohere, Anthropic, Cloudflare, and DeepSeek, enabling easy use of Mixture of Experts. The software is self-hosted, Apache 2.0 licensed, and provides a unified endpoint for managing requests and responses across providers.
Key features
- Unified, OpenAI-compatible API endpoint supporting multiple providers (OpenAI, Anthropic, Groq, Cohere, Ollama, and more)
- Tool-use support with function calling across supported providers - Full Model Context Protocol (MCP) integration for automatic tool discovery and exposure - Agent-to-Agent (A2A) server for relaying calls through auth, guardrails, and telemetry - OPA/Rego guardrails with secret and PII detection, and optional external guardrail service - Streaming, vision/multimodal support, environment configuration, Docker & Compose, Kubernetes-ready, OpenTelemetry metrics, enterprise features (OIDC auth, TLS), lightweight (~10MB), privacy-first, and CLI tool
Use cases
Development teams can deploy the gateway as a self-hosted, privacy-first proxy to access multiple LLMs through a single endpoint, simplifying integration and testing. Production environments can leverage horizontal scaling, Kubernetes deployment, and guardrails for secure, auditable LLM access. Organizations can use MCP and A2A servers to build agent workflows with automatic tool discovery and inter-agent communication. The middleware bypass mechanisms give fine-grained control over gateway capabilities per request.
Getting started
See the repository README for installation instructions.
License and pricing
Licensed under Apache 2.0. The software is described as open source with privacy-first, zero data collection, and Apache 2.0 licensing.
Based on the project's public README and repository metadata.