semantic-router
An open, programmable decision layer for models and compute.
What is semantic-router?
An open, programmable decision layer for models and compute.
Overview of semantic-router
What it is
semantic-router is an open, programmable decision layer for models and compute. It provides a single API that selects or combines models for each call, guided by user-defined policy. The router works across local, private, and cloud compute backends, allowing agent harnesses to maintain their own agent loop, tools, and task state while the router handles model selection and backend routing.
Key features
- Selects or combines models per call based on policy.
- Supports multiple backends across local, private, and cloud compute.
- Allows setting priorities for quality, latency, and cost.
- Keeps calls within approved geographic locations.
- Provides a single API for many models.
- Integrates with existing agent harnesses without replacing them.
Use cases
semantic-router is used in scenarios where different models excel at different tasks, hardware varies in speed and capacity, and compute must be distributed across edge, private, and cloud locations. It is suitable for production LLM inference systems that need dynamic model selection, mixture-of-models architectures, and policy-driven routing. Example applications include real-time hallucination detection, cursor-style auto model selection, and category-aware semantic caching.
Getting started
bashcurl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel stableFor pip, uv, or agent-driven installation, see the repository README.
License and pricing
This project is licensed under Apache-2.0 and is open source. Pricing is not stated.
Based on the project's public README and repository metadata.