ome
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
What is ome?
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
Overview of ome
What it is
OME (Open Model Engine) is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs). It automates model management, intelligent runtime selection, resource utilization, and deployment patterns.
Key features
- Model Management: Models are first-class custom resources with automated parsing of architecture, parameter count, and capabilities from model files. Supports distributed storage, double encryption, namespace scoping, and formats including SafeTensors, PyTorch, TensorRT, and ONNX.
- Intelligent Runtime Selection: Automatic matching of models to optimal runtime configurations through weighted scoring based on architecture, format, quantization, parameter size, and framework compatibility.
- Optimized Deployments: Supports prefill-decode disaggregation, multi-node inference, and traditional Kubernetes deployments with canary and blue-green rollout strategies and advanced scaling controls.
- Resource Optimization: Specialized GPU bin-packing scheduling with dynamic re-optimization to maximize cluster efficiency while ensuring high availability.
- Runtime Integrations: First-class support for SGLang (cache-aware load balancing, multi-node deployment, prefill-decode disaggregated serving, multi-LoRA adapter serving) and vLLM for high-throughput inference.
- Accelerator Management: Hardware-aware scheduling through AcceleratorClass resources defining GPU capabilities, discovery patterns, and cost information with policies like BestFit, Cheapest, or MostCapable.
- Fleet Capacity and GPU Operations (alpha): Optional add-on control planes including quota manager (assembles AcceleratorQuota tree into Kueue) and Alfred (GPU cluster caretaker, recommend-only by default).
Use cases
Enterprise deployment and operation of LLMs on Kubernetes. Automating model lifecycle management, optimizing GPU resource utilization across clusters, implementing advanced serving patterns like prefill-decode disaggregation and multi-node inference, and managing fleet-level GPU capacity and operations.
Getting started
Requires Kubernetes 1.28 or newer.Install OME from the OCI registry:helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --namespace ome --create-namespacehelm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources --namespace omeInstall the kubectl plugin:kubectl krew install --manifest-url https://github.com/ome-projects/ome/releases/latest/download/ome.yaml
License and pricing
License: Apache-2.0. Pricing: open source.
這是什麼
OME (Open Model Engine) 是一個 Kubernetes 運算子,用於大型語言模型 (LLM) 的企業級管理與服務。它自動化模型管理、智慧執行時期選擇、資源使用率與部署模式。
主要功能
- 模型管理:模型為一等自訂資源,可自動解析模型檔案的架構、參數數量與能力。支援分散式儲存、雙重加密、命名空間範圍,以及 SafeTensors、PyTorch、TensorRT 與 ONNX 等格式。
- 智慧執行時期選擇:透過基於架構、格式、量化、參數大小與框架相容性的加權評分,自動將模型配對至最佳執行時期設定。
- 最佳化部署:支援預填充-解碼分離、多節點推論與傳統 Kubernetes 部署,具備金絲雀與藍綠部署策略及進階擴展控制。
- 資源最佳化:專用 GPU 裝箱排程,具備動態重新最佳化,以最大化叢集效率並確保高可用性。
- 執行時期整合:一級支援 SGLang(快取感知負載平衡、多節點部署、預填充-解碼分離服務、多 LoRA 適配器服務)與 vLLM 高吞吐量推論。
- 加速器管理:透過 AcceleratorClass 資源進行硬體感知排程,定義 GPU 能力、發現模式與成本資訊,支援 BestFit、Cheapest 或 MostCapable 等策略。
- 叢集容量與 GPU 操作(alpha):選用附加控制平面,包括配額管理器(將 AcceleratorQuota 樹組裝至 Kueue)與 Alfred(GPU 叢集管理員,預設僅建議)。
使用情境
在 Kubernetes 上進行 LLM 的企業級部署與操作。自動化模型生命週期管理、最佳化跨叢集 GPU 資源使用率、實作預填充-解碼分離與多節點推論等進階服務模式,以及管理叢集層級的 GPU 容量與操作。
快速開始
需要 Kubernetes 1.28 或更新版本。從 OCI 註冊表安裝 OME:helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --namespace ome --create-namespacehelm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources --namespace ome安裝 kubectl 外掛:kubectl krew install --manifest-url https://github.com/ome-projects/ome/releases/latest/download/ome.yaml
授權與價格
授權:Apache-2.0。價格:開源。
Based on the project's public README and repository metadata.