tair-kvcache
Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.
What is tair-kvcache?
Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.
Overview of tair-kvcache
What it is
Tair KVCache is Alibaba Cloud's high-performance KVCache system designed for Large Language Model (LLM) inference scenarios. It uses technologies such as distributed memory pooling and dynamic multi-level caching to achieve acceleration and efficiency improvement while reducing resource costs. The open-sourced components include the global KVCache management system Tair KVCache Manager and the LLM inference simulation system Tair KVCache HiSim.
Key features
- Tair KVCache Manager provides unified KVCache metadata management services, including global metadata management, KVCache queries, and storage capacity management.
- Supports multiple matching logics: prefix matching, sliding window matching, KV matching, etc.
- Implements a two-phase write mechanism for data reliability.
- Compatible with multiple storage backends (HF3FS, Mooncake, NFS, etc.) and dynamically selects storage backends based on availability and water levels.
- Includes a cache simulation and optimization component (Optimizer) that replays KVCache access traces to simulate behavior and guide parameter optimization.
- Tair KVCache HiSim is a high-performance CPU-based simulation system for LLM inference, enabling fast, low-cost, and high-fidelity prediction of key performance metrics (TTFT, TPOT, throughput) without requiring GPU resources.
- HiSim currently supports SGLang v0.5.6.post2 with Qwen3 Dense series models on H20 GPUs, achieving prediction errors below 5%.
Use cases
- Global KVCache metadata management for LLM inference clusters.
- Accelerating LLM inference through distributed memory pooling and multi-level caching.
- Simulating and optimizing LLM inference performance (e.g., TTFT, TPOT, throughput) by replaying real-world workload traces.
- Evaluating different models, hardware, inference engines, and configurations without actual GPU resources.
Getting started
See the repository README for installation and quick-start instructions.
License and pricing
Licensed under Apache-2.0. Open source.
Based on the project's public README and repository metadata.