kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
What is kserve?
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
Overview of kserve
What it is
KServe is a standardized distributed generative and predictive AI inference platform for scalable, multi-framework deployment on Kubernetes. It is a Cloud Native Computing Foundation (CNCF) incubating project.
Key features
- Generative AI: Optimized backends for vLLM and llm-d, OpenAI-compatible inference protocol, GPU acceleration, model caching, KV cache offloading, request-based autoscaling, and native Hugging Face support.
- Predictive AI: Multi-framework support (TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, etc.), intelligent routing between predictor/transformer/explainer components, advanced deployments (canary rollouts, inference pipelines, ensembles with InferenceGraph), request-based autoscaling with scale-to-zero, built-in model explainability, advanced monitoring (payload logging, outlier/adversarial/drift detection), and cost efficiency via scale-to-zero.
Use cases
KServe is used for deploying and serving both generative AI models (e.g., large language models) and predictive AI models (e.g., classification, regression) on Kubernetes. It supports scenarios requiring high-performance GPU serving, model caching, canary rollouts, inference pipelines, and scale-to-zero for cost savings.
Getting started
See the repository README for installation options: standalone (standard Kubernetes, Knative serverless, ModelMesh high-scale) or as part of Kubeflow. Quick installation instructions are available in the KServe documentation.
License and pricing
KServe is open source under the Apache-2.0 license.
Based on the project's public README and repository metadata.