LMCache
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
What is LMCache?
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
Overview of LMCache
What it is
LMCache is a KV cache management layer for LLM inference. It transforms KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality. LMCache reduces time-to-first-token (TTFT) and improves throughput, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG). It is vendor-neutral and can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers.
Key features
- Engine-independent deployment: LMCache runs as a standalone daemon process, managing KV cache independently from the inference engine process, so KV cache is not lost if the inference engine crashes (no fate-sharing).
- Persistent, tiered KV cache offloading and reuse: Moves KV caches out of GPU memory into a tiered storage hierarchy (CPU memory, local storage, remote backends), enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT.
- Production-level KV cache observability: Provides rich metrics including Kubernetes health monitoring, performance diagnostics, KV-cache-specific metrics (request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance), and management metrics (user-specific usage).
- Pluggable storage and transport backends: Integrates remote storage and KV transfer backends through a unified interface, supporting CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS.
- Non-prefix KV reuse: Reuses cached KV blocks at any position in the prompt, extending beyond prefix caching, using CacheBlend to selectively recompute tokens for quality recovery.
- PD disaggregation and KV transfer: Supports KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL.
- Pluggable KV transformation: Provides a simple interface for researchers to implement compression, token dropping, and custom serialization via a flexible SERDE interface.
Use cases
LMCache is designed for long-context agentic workloads, multi-turn conversations, and knowledge-augmented applications such as RAG. It reduces TTFT and improves throughput by reusing KV caches across requests and sessions, making it suitable for production LLM serving environments that require low latency and high efficiency.
Getting started
Install LMCache via pip: bash pip install lmcache For more setup options, examples, and deployment guides, see the repository README.
License and pricing
LMCache is licensed under Apache License 2.0 and is open source. No pricing information is stated beyond the license.
Based on the project's public README and repository metadata.