lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

What is lmdeploy?

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Overview of lmdeploy

What it is

LMDeploy is a toolkit for compressing, deploying, and serving large language models (LLMs). It is developed by the MMRazor and MMDeploy teams. The project provides two inference engines: TurboMind, which focuses on performance optimization, and PyTorch, which is developed entirely in Python to lower barriers for developers.

Key features

  • Efficient Inference: Delivers up to 1.8x higher request throughput than vLLM, using persistent batch, blocked KV cache, dynamic split&fuse, tensor parallelism, and high-performance CUDA kernels.
  • Effective Quantization: Supports weight-only and k/v quantization; 4-bit inference performance is 2.4x higher than FP16. Quantization quality has been confirmed via OpenCompass evaluation.
  • Effortless Distribution Server: A request distribution service enables easy deployment of multi-model services across multiple machines and cards.
  • Excellent Compatibility: Supports KV Cache Quant, AWQ, and Automatic Prefix Caching to be used simultaneously.

Use cases

LMDeploy is suitable for compressing, deploying, and serving LLMs in production environments. It can be used for offline inference pipelines, online serving, and multi-model, multi-machine, multi-card inference services. It supports a wide range of LLMs and VLMs, including Llama, InternLM, Qwen, DeepSeek, and many others.

Getting started

Install LMDeploy using pip in a conda environment (Python 3.10–3.13):shellconda create -n lmdeploy python=3.12 -yconda activate lmdeploypip install lmdeploy

License and pricing

LMDeploy is open source under the Apache-2.0 license. Pricing is not stated; the project is listed as open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/internlmlmdeploy.svg)](https://yunodb.com/tools/internlmlmdeploy?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/internlmlmdeploy?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/internlmlmdeploy.svg" alt="Featured on YunoDB" height="20"></a>