omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
What is omlx?
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Overview of omlx
What it is
oMLX is an LLM inference server optimized for Apple Silicon, featuring continuous batching and tiered KV caching (hot RAM + cold SSD). It is managed from the macOS menu bar. The server supports text LLMs, vision-language models (VLM), OCR models, embeddings, and rerankers. It provides an OpenAI-compatible API and a built-in chat UI.
Key features
- Continuous batching for concurrent requests via mlx-lm's BatchGenerator.
- Tiered KV cache with hot (RAM) and cold (SSD) tiers; blocks are offloaded to SSD in safetensors format and restored on matching prefix requests, even after server restart.
- macOS menu bar management with an admin dashboard (web UI at /admin) for real-time monitoring, model management, chat, benchmark, and per-model settings.
- Supports text LLMs, vision-language models (VLM), OCR models, embeddings, and rerankers on Apple Silicon.
- Experimental multi-Mac inference (source builds) using MLX pipeline ranks over Ring or Thunderbolt RDMA/JACCL, with a Prefill Accelerator feature.
- Optional Affine4 and Affine8 KV compression with Metal TensorOps decode attention and a portable fallback.
- Claude Code optimization: reports the model's real context window to auto-compact and uses SSE keep-alive to prevent read timeouts.
Use cases
- Running local LLMs for coding work with tools like Claude Code, benefiting from persistent KV cache across context changes.
- Deploying vision-language models with continuous batching and tiered caching on a single Mac.
- Managing multiple models from the macOS menu bar with per-model settings and real-time monitoring.
- Experimental distributed inference across multiple Macs for larger models.
Getting started
bash # macOS App: Download .dmg from Releases, drag to Applications. # Homebrew: brew tap jundot/omlx https://github.com/jundot/omlx brew install jundot/omlx/omlx --with-custom-kernel # Quickstart CLI (after install): omlx start omlx serve --model-dir ~/models
License and pricing
License: Apache-2.0. Pricing: Open source (free).
Based on the project's public README and repository metadata.