mlc-llm
Universal LLM Deployment Engine with ML Compilation
What is mlc-llm?
Universal LLM Deployment Engine with ML Compilation
Overview of mlc-llm
What it is
MLC LLM is a machine learning compiler and high-performance deployment engine for large language models. Its mission is to enable everyone to develop, optimize, and deploy AI models natively on everyone's platforms. It compiles and runs code on MLCEngine, a unified high-performance LLM inference engine that supports multiple platforms.
Key features
- ML compilation for large language models
- High-performance deployment engine
- Supports multiple GPU backends: Vulkan, ROCm, CUDA, Metal, OpenCL
- Runs on Linux, Windows, macOS, Web browsers (WebGPU and WASM), iOS/iPadOS, and Android
- Provides OpenAI-compatible API via REST server, Python, JavaScript, iOS, and Android
- Unified engine across all supported platforms
Use cases
- Developing and optimizing large language models
- Deploying LLMs natively on desktop, mobile, web, and server platforms
- Running inference on AMD, NVIDIA, Apple, and Intel GPUs
- Building applications that require an OpenAI-compatible API endpoint
Getting started
See the repository README for installation and quick start instructions.
License and pricing
MLC LLM is open source under the Apache-2.0 license.
Based on the project's public README and repository metadata.