kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

What is kserve?

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

Overview of kserve

What it is

KServe is a standardized distributed generative and predictive AI inference platform for scalable, multi-framework deployment on Kubernetes. It is a Cloud Native Computing Foundation (CNCF) incubating project.

Key features

  • Generative AI: Optimized backends for vLLM and llm-d, OpenAI-compatible inference protocol, GPU acceleration, model caching, KV cache offloading, request-based autoscaling, and native Hugging Face support.
  • Predictive AI: Multi-framework support (TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, etc.), intelligent routing between predictor/transformer/explainer components, advanced deployments (canary rollouts, inference pipelines, ensembles with InferenceGraph), request-based autoscaling with scale-to-zero, built-in model explainability, advanced monitoring (payload logging, outlier/adversarial/drift detection), and cost efficiency via scale-to-zero.

Use cases

KServe is used for deploying and serving both generative AI models (e.g., large language models) and predictive AI models (e.g., classification, regression) on Kubernetes. It supports scenarios requiring high-performance GPU serving, model caching, canary rollouts, inference pipelines, and scale-to-zero for cost savings.

Getting started

See the repository README for installation options: standalone (standard Kubernetes, Knative serverless, ModelMesh high-scale) or as part of Kubeflow. Quick installation instructions are available in the KServe documentation.

License and pricing

KServe is open source under the Apache-2.0 license.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/kservekserve.svg)](https://yunodb.com/tools/kservekserve?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/kservekserve?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/kservekserve.svg" alt="Featured on YunoDB" height="20"></a>