vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

What is vllm?

A high-throughput and memory-efficient inference and serving engine for LLMs

Overview of vllm

What it is

vLLM is a fast and easy-to-use library for LLM inference and serving. Originally developed in the Sky Computing Lab at UC Berkeley, it is an open-source project maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.

Key features

  • State-of-the-art serving throughput with efficient management of attention key and value memory using PagedAttention
  • Continuous batching, chunked prefill, prefix caching, and speculative decoding (n-gram, suffix, EAGLE, DFlash)
  • Fast model execution with piecewise and full CUDA/HIP graphs, and quantization support (FP8, INT8, INT4, GPTQ/AWQ, GGUF, and more)
  • Optimized attention kernels (FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, Triton) and GEMM/MoE kernels using CUTLASS, TRTLLM-GEN, CuTeDSL
  • Seamless integration with 200+ Hugging Face model architectures, including decoder-only, MoE, hybrid, multi-modal, embedding, and reward models
  • OpenAI-compatible API server, plus Anthropic Messages API and gRPC support; streaming outputs; structured output generation; tool calling and reasoning parsers
  • Distributed inference with tensor, pipeline, data, expert, and context parallelism; supports NVIDIA, AMD, Intel GPUs, x86/ARM/PowerPC CPUs, and diverse hardware plugins (Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, etc.)

Use cases

vLLM is used for high-throughput LLM serving in production environments, enabling efficient inference for large language models across various hardware. It supports applications such as chatbots, code generation, multi-modal reasoning, and retrieval-augmented generation, with the ability to handle structured outputs and tool calling.

Getting started

bashuv pip install vllmvllm download-kernels # precompiled kernels for faster startup

License and pricing

vLLM is open source under the Apache-2.0 license. Pricing is not stated; it is free to use under the license terms.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/vllm-projectvllm.svg)](https://yunodb.com/tools/vllm-projectvllm?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/vllm-projectvllm?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/vllm-projectvllm.svg" alt="Featured on YunoDB" height="20"></a>