inference
Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous har
What is inference?
Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.
Overview of inference
What it is
Xorbits Inference (Xinference) is a library designed to serve language, speech recognition, and multimodal models. It allows deployment and serving of built-in models using a single command. It supports OpenAI-compatible RESTful API, RPC, CLI, and WebUI. It can run on cloud, on-prem, or local hardware.
Key features
- Model serving simplified: deploy and serve large language, speech recognition, and multimodal models with a single command.
- State-of-the-art built-in models: access to cutting-edge open-source models.
- Heterogeneous hardware utilization: uses GPUs and CPUs via ggml.
- Flexible API and interfaces: OpenAI-compatible RESTful API (including Function Calling), RPC, CLI, and WebUI.
- Distributed deployment: distribute model inference across multiple devices or machines.
- Built-in integration with third-party libraries: LangChain, LlamaIndex, Dify, Chatbox.
- Supports multiple inference engines: vLLM, GGML, TensorRT.
- Supports multiple platforms: CPU, Metal.
- Supports image models (text-to-image), text embedding models, multimodal models, audio models.
- More OpenAI functionalities: Function Calling.
Use cases
- Serving large language, speech recognition, and multimodal models for experimentation and production.
- Distributed inference across multiple nodes.
- Integration with AI application platforms like Dify, FastGPT, RAGFlow, MaxKB, and Xagent for building RAG pipelines, AI assistants, and agent workflows.
- Suitable for researchers, developers, and data scientists.
Getting started
Install via pip: bash pip install xinference xinference-local One-command installer: Linux/macOS: bash curl -fsSL https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.sh | sh Windows PowerShell: powershell irm https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.ps1 | iex Docker and Kubernetes deployment options are also available; see the repository README for details.
License and pricing
The project is open source under the Apache-2.0 license. A commercial edition, Xinference Enterprise, is available for teams running models in production, with options for self-hosted or managed cloud deployment.
Based on the project's public README and repository metadata.