vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
What is vllm?
A high-throughput and memory-efficient inference and serving engine for LLMs
Overview of vllm
What it is
vLLM is a fast and easy-to-use library for LLM inference and serving. Originally developed in the Sky Computing Lab at UC Berkeley, it is an open-source project maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.
Key features
- State-of-the-art serving throughput with efficient management of attention key and value memory using PagedAttention
- Continuous batching, chunked prefill, prefix caching, and speculative decoding (n-gram, suffix, EAGLE, DFlash)
- Fast model execution with piecewise and full CUDA/HIP graphs, and quantization support (FP8, INT8, INT4, GPTQ/AWQ, GGUF, and more)
- Optimized attention kernels (FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, Triton) and GEMM/MoE kernels using CUTLASS, TRTLLM-GEN, CuTeDSL
- Seamless integration with 200+ Hugging Face model architectures, including decoder-only, MoE, hybrid, multi-modal, embedding, and reward models
- OpenAI-compatible API server, plus Anthropic Messages API and gRPC support; streaming outputs; structured output generation; tool calling and reasoning parsers
- Distributed inference with tensor, pipeline, data, expert, and context parallelism; supports NVIDIA, AMD, Intel GPUs, x86/ARM/PowerPC CPUs, and diverse hardware plugins (Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, etc.)
Use cases
vLLM is used for high-throughput LLM serving in production environments, enabling efficient inference for large language models across various hardware. It supports applications such as chatbots, code generation, multi-modal reasoning, and retrieval-augmented generation, with the ability to handle structured outputs and tool calling.
Getting started
bashuv pip install vllmvllm download-kernels # precompiled kernels for faster startup
License and pricing
vLLM is open source under the Apache-2.0 license. Pricing is not stated; it is free to use under the license terms.
Based on the project's public README and repository metadata.