gateway
A blazing fast AI Gateway with integrated guardrails. Route to 1,600+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
What is gateway?
A blazing fast AI Gateway with integrated guardrails. Route to 1,600+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
Overview of gateway
What it is
The AI Gateway is a lightweight, open-source, and enterprise-ready solution designed for fast, reliable, and secure routing to over 1,600 language, vision, audio, and image models. It provides a single, OpenAI-compatible API that allows integration with any language model in under 2 minutes. The gateway supports 45+ providers and 8+ agent frameworks, and includes integrated guardrails for input/output verification.
Key features
- Fallbacks and automatic retries (up to 5 attempts with exponential backoff) for improved reliability
- Load balancing across multiple API keys or providers with configurable weights
- Multi-modal support for vision, audio (text-to-speech & speech-to-text), and image generation models using the OpenAI signature
- 40+ pre-built guardrails for input and output verification, with support for custom guardrails
- Secure key management with virtual keys and role-based access control (RBAC)
- Smart caching (simple and semantic) to reduce costs and latency
- Usage analytics for monitoring request volume, latency, costs, and error rates
Use cases
- Routing requests to multiple LLMs and providers from a single API endpoint
- Implementing safety and compliance guardrails on LLM inputs and outputs
- Managing costs through caching, retries, and provider optimization
- Integrating with agent frameworks (Autogen, CrewAI, LangChain, LlamaIndex, etc.) for complex AI workflows
- Deploying in enterprise environments with private cloud options (AWS, Azure, GCP, Kubernetes)
Getting started
npx @portkey-ai/gateway
The gateway runs on http://localhost:8787/v1 and the console on http://localhost:8787/public/. For deployment guides (Docker, Node.js, Cloudflare, etc.) see the repository README.
License and pricing
MIT License. The project is open source.
Based on the project's public README and repository metadata.