laya-mlx
Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.
What is laya-mlx?
Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.
Overview of laya-mlx
What it is
Laya-MLX is a native MLX runtime for Laya typed decision models. It runs on Apple Silicon without PyTorch, Transformers, or a cloud API. It answers constrained questions (pick an option, score against a rubric, judge true or false) with calibrated probabilities from one bidirectional forward pass. It generates no tokens and returns no JSON to parse.
Key features
- Runs Laya weights in MLX, without PyTorch, Transformers, or a cloud API.
- Supports three question types:
choice(probability per option),score(probability per level and expected level), andnoul(P(true)). - Includes language routing, batch prediction, confidence gate, large choice set handling (shortlist and tournament), and throughput options (compile, pad_to_multiple, cache_prompts).
- Provides CLI, HTTP server, MCP server, evaluation and calibration tools, and integrations with LangChain, LlamaIndex, CrewAI, and Agno (as extras).
- Export functionality to write a standalone MLX checkpoint.
- Performance: 13.4 ms median for a short English decision, 7.4 ms with the multilingual checkpoint on M3 Max (FP16).
Use cases
- Routing customer requests to the correct department based on a description of options.
- Determining whether a customer asks for a refund (yes/no).
- Scoring responses against ordered rubric levels.
- Any constrained decision task that benefits from calibrated probabilities without text generation.
Getting started
Requires Apple Silicon, macOS 14+ and Python 3.11+.
pip install laya-mlx
See the repository README for a full usage example.
License and pricing
Apache-2.0 license, open source.
Based on the project's public README and repository metadata.