llm-d

State of the art inference performance with modern accelerators on Kubernetes

What is llm-d?

State of the art inference performance with modern accelerators on Kubernetes

Overview of llm-d

What it is

llm-d is a high-performance distributed inference serving stack optimized for production deployments on Kubernetes. It is a Cloud Native Computing Foundation (CNCF) sandbox project, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The project aims to achieve state-of-the-art inference performance across various hardware accelerators and infrastructure providers.

Key features

  • Intelligent Routing: Maximizes performance with prefix-cache and load-aware balancing, including experimental predicted latency-based scheduling.
  • Advanced KV-Cache Management: Increases effective working set size for multi-turn requests with tiered offloading to CPU or disk and precise global indexing of KV cache state.
  • Serving Large Models: Optimizes massive models using prefill/decode disaggregation and wide expert-parallelism over fast accelerator interconnects.
  • Agent-Aware Serving: Serves multi-turn agent workloads natively with session-aware routing, turn-priority scheduling, and native Responses API state.
  • Operational Excellence: Provides multi-tenant quality of service, zero-downtime operations, SLO-aware autoscaling, and end-to-end observability.
  • Batch Processing: Manages large-scale offline inference with OpenAI-compatible Batch APIs and asynchronous processing.

Use cases

Production inference serving on Kubernetes with multi-tenant quality of service. Serving large language models across different hardware accelerators (NVIDIA, AMD, Intel GPUs, Google TPUs). Handling agent workloads with session-aware routing. Batch processing of offline inference tasks. Achieving high throughput and low latency for real-world traffic.

Getting started

See the repository README for installation and quick-start instructions.

License and pricing

This project is licensed under Apache License 2.0. It is open source with no stated pricing.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/llm-dllm-d.svg)](https://yunodb.com/tools/llm-dllm-d?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/llm-dllm-d?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/llm-dllm-d.svg" alt="Featured on YunoDB" height="20"></a>