llm-d
State of the art inference performance with modern accelerators on Kubernetes
What is llm-d?
State of the art inference performance with modern accelerators on Kubernetes
Overview of llm-d
What it is
llm-d is a high-performance distributed inference serving stack optimized for production deployments on Kubernetes. It is a Cloud Native Computing Foundation (CNCF) sandbox project, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The project aims to achieve state-of-the-art inference performance across various hardware accelerators and infrastructure providers.
Key features
- Intelligent Routing: Maximizes performance with prefix-cache and load-aware balancing, including experimental predicted latency-based scheduling.
- Advanced KV-Cache Management: Increases effective working set size for multi-turn requests with tiered offloading to CPU or disk and precise global indexing of KV cache state.
- Serving Large Models: Optimizes massive models using prefill/decode disaggregation and wide expert-parallelism over fast accelerator interconnects.
- Agent-Aware Serving: Serves multi-turn agent workloads natively with session-aware routing, turn-priority scheduling, and native Responses API state.
- Operational Excellence: Provides multi-tenant quality of service, zero-downtime operations, SLO-aware autoscaling, and end-to-end observability.
- Batch Processing: Manages large-scale offline inference with OpenAI-compatible Batch APIs and asynchronous processing.
Use cases
Production inference serving on Kubernetes with multi-tenant quality of service. Serving large language models across different hardware accelerators (NVIDIA, AMD, Intel GPUs, Google TPUs). Handling agent workloads with session-aware routing. Batch processing of offline inference tasks. Achieving high throughput and low latency for real-world traffic.
Getting started
See the repository README for installation and quick-start instructions.
License and pricing
This project is licensed under Apache License 2.0. It is open source with no stated pricing.
Based on the project's public README and repository metadata.