trainer
Distributed AI Model Training and LLM Fine-Tuning on Kubernetes
What is trainer?
Distributed AI Model Training and LLM Fine-Tuning on Kubernetes
Overview of trainer
What it is
Kubeflow Trainer is a Kubernetes-native distributed AI platform for scalable large language model (LLM) fine-tuning, hyperparameter optimization, and training of AI models across a wide range of frameworks, including PyTorch, MLX, HuggingFace, DeepSpeed, Megatron-LM, JAX, XGBoost, and more. It brings MPI to Kubernetes, orchestrating multi-node, multi-GPU distributed jobs efficiently across high-performance computing (HPC) clusters. The project is currently in alpha status.
Key features
- Supports multiple AI frameworks: PyTorch, MLX, HuggingFace, DeepSpeed, Megatron-LM, JAX, XGBoost, and more.
- MPI integration for multi-node, multi-GPU distributed training on Kubernetes.
- Integrates with Kueue for topology-aware scheduling and multi-cluster job dispatching, Slurm Bridge for hybrid Kubernetes/Slurm clusters, and KAI Scheduler for GPU-aware scheduling.
- Reuses Kubernetes-native building blocks like JobSet and LeaderWorkerSet for workload orchestration.
- Provides a distributed data cache for streaming large-scale data with zero-copy transfer to GPU nodes.
- Offers a Python SDK with TrainJob and Runtime APIs for developing and fine-tuning LLMs.
Use cases
- Scalable fine-tuning of large language models (LLMs) on Kubernetes.
- Hyperparameter optimization for AI models.
- Distributed training across multiple frameworks and hardware configurations.
- High-performance computing (HPC) workloads requiring MPI communication between GPU nodes.
Getting started
To deploy the latest stable release, run the following Helm command: bash helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer --namespace kubeflow-system --create-namespace --set runtimes.defaultEnabled=true For full installation and usage instructions, refer to the official Kubeflow Trainer documentation.
License and pricing
The project is licensed under Apache-2.0 and is open source (free to use).
Based on the project's public README and repository metadata.