zml
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild
What is zml?
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild
Overview of zml
What it is
ZML is a production inference stack built with Zig, MLIR, and Bazel. It is designed to decouple AI workloads from proprietary hardware, allowing the same codebase to run on multiple accelerators without rewriting.
Key features
- Supports NVIDIA, AMD, Intel, Google TPU, and AWS Trainium/Inferentia 2 accelerators.
- Compiled directly to each target hardware for peak performance.
- Supports models including Llama 3.1/3.2, Qwen 3.5, and LFM 2.5.
- Provides a unified LLM CLI for loading models from Hugging Face, local directories, or S3.
- Includes examples for MNIST, sharding, I/O, and benchmarking.
Use cases
- Running large language models (LLMs) on various hardware platforms.
- Deploying AI inference workloads across different accelerators without code changes.
- Experimenting with model inference on CPU, GPU, or TPU.
- Benchmarking model loading and execution performance.
Getting started
Prerequisites: Install Bazel (recommended via Bazelisk). On macOS: bash brew install bazelisk On Linux: bash curl -L -o /usr/local/bin/bazel 'https://github.com/bazelbuild/bazelisk/releases/download/v1.28.0/bazelisk-linux-amd64' chmod +x /usr/local/bin/bazel Run the MNIST smoke test: bash bazel run //examples/mnist For LLM quickstart, authenticate with Hugging Face if needed: bash bazel run //tools/hf -- auth login Then run a prompt: bash bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt='What is the capital of France?' To run on GPU/TPU, append platform flags, e.g., for CUDA: bash bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt='Write a haiku about Zig'
License and pricing
ZML is licensed under the Apache 2.0 license. It is open source.
Based on the project's public README and repository metadata.