tair-kvcache

Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.

What is tair-kvcache?

Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.

Overview of tair-kvcache

What it is

Tair KVCache is Alibaba Cloud's high-performance KVCache system designed for Large Language Model (LLM) inference scenarios. It uses technologies such as distributed memory pooling and dynamic multi-level caching to achieve acceleration and efficiency improvement while reducing resource costs. The open-sourced components include the global KVCache management system Tair KVCache Manager and the LLM inference simulation system Tair KVCache HiSim.

Key features

  • Tair KVCache Manager provides unified KVCache metadata management services, including global metadata management, KVCache queries, and storage capacity management.
  • Supports multiple matching logics: prefix matching, sliding window matching, KV matching, etc.
  • Implements a two-phase write mechanism for data reliability.
  • Compatible with multiple storage backends (HF3FS, Mooncake, NFS, etc.) and dynamically selects storage backends based on availability and water levels.
  • Includes a cache simulation and optimization component (Optimizer) that replays KVCache access traces to simulate behavior and guide parameter optimization.
  • Tair KVCache HiSim is a high-performance CPU-based simulation system for LLM inference, enabling fast, low-cost, and high-fidelity prediction of key performance metrics (TTFT, TPOT, throughput) without requiring GPU resources.
  • HiSim currently supports SGLang v0.5.6.post2 with Qwen3 Dense series models on H20 GPUs, achieving prediction errors below 5%.

Use cases

  • Global KVCache metadata management for LLM inference clusters.
  • Accelerating LLM inference through distributed memory pooling and multi-level caching.
  • Simulating and optimizing LLM inference performance (e.g., TTFT, TPOT, throughput) by replaying real-world workload traces.
  • Evaluating different models, hardware, inference engines, and configurations without actual GPU resources.

Getting started

See the repository README for installation and quick-start instructions.

License and pricing

Licensed under Apache-2.0. Open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/alibabatair-kvcache.svg)](https://yunodb.com/tools/alibabatair-kvcache?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/alibabatair-kvcache?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/alibabatair-kvcache.svg" alt="Featured on YunoDB" height="20"></a>