sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
What is sglang?
SGLang is a high-performance serving framework for large language models and multimodal models.
Overview of sglang
What it is
SGLang is an open-source inference framework for large language, vision-language, and diffusion models. It is optimized for agentic workloads, large-scale serving, and RL rollouts. SGLang Diffusion, a built-in image and video generation engine, is included in the same repository and Python package.
Key features
- Open-source inference framework supporting large language, vision-language, and diffusion models.
- Optimized for agentic workloads, large-scale serving, and reinforcement learning rollouts.
- Built-in image and video generation engine (SGLang Diffusion).
- Supports a wide range of hardware: NVIDIA (B200, H100, A100, RTX series, etc.), AMD (Instinct MI300X, etc.), Google TPU (v6e, v7), Intel (Arc GPUs, Xeon CPUs), Apple Silicon (Macs via Metal/MLX), Huawei Ascend (A2, A3, etc.), and Moore Threads (MTT S5000).
- TPU serving via SGLang-TorchTPU (PyTorch-based) and SGL-JAX (JAX-based).
- Ecosystem includes audio model serving (SGLang Omni), speculative decoding (SpecForge), RL training frameworks, and deployment tools.
Use cases
- Serving large language models, vision-language models, and diffusion models for production workloads.
- Agentic workloads requiring fast inference and tool use.
- Large-scale serving with routing, load balancing, and cluster orchestration.
- Reinforcement learning rollout generation.
- Image and video generation using diffusion models.
- Audio model serving for text-to-speech (TTS) and automatic speech recognition (ASR) via SGLang Omni.
- Speculative decoding with trained draft models via SpecForge.
Getting started
Pull the Docker image:bashdocker pull lmsysorg/sglang:latestAlternatively, install SGLang in an activated Python environment with uv:bashuv pip install --prerelease=allow sglangAfter installation, launch your model. See the repository README for quickstart and cookbook instructions.
License and pricing
License: Apache-2.0. Pricing: open source (free).
Based on the project's public README and repository metadata.