BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, an
What is BigMoeOnEdge?
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
Overview of BigMoeOnEdge
What it is
BigMoeOnEdge is an adaptive streaming optimizer for Mixture-of-Experts (MoE) models. It enables running MoE models that are several times larger than the available RAM on edge devices such as phones, laptops, and single-board computers. The project is built on top of llama.cpp's public API and provides a local OpenAI-compatible server. It works on Android, macOS, Windows, and Linux.
Key features
- Adaptive streaming optimizer that measures the machine (memory, storage speed, CPU, GPU, NPU) and builds an optimal plan for that exact device, showing the reason behind every choice.
- Compatibility with any hardware, any OS, any MoE gguf model, and any quantization supported by llama.cpp.
- Local OpenAI-compatible server, allowing chat clients and coding agents to run fully locally and devices to share models over Wi-Fi.
- Real-time model header inspection: the engine reads the model's header before download and tells how it will run.
- Planned streaming is lossless: output is identical to running the model fully in memory, verified by tests.
Use cases
Running large MoE models on memory-constrained edge devices for local AI agents, chatbots, and coding assistants. Suitable for users who want to run frontier-size models on a phone, laptop, or edge hardware without sacrificing output quality. Also useful for developers who need a local OpenAI-compatible API for their tools.
Getting started
bashbmoe-cli -m model.gguf --auto -p 'Explain MoE routing.' # measure, plan, runbmoe-cli -m model.gguf --plan # show the plan and exitOr, for desktop: download the bmoe-desktop archive from the latest release and run bmoe-server --open. For Android: install the APK from the latest release.
License and pricing
Apache-2.0. The project is open source.
Based on the project's public README and repository metadata.
