llama-modchip
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
What is llama-modchip?
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
Overview of llama-modchip
What it is
llama-modchip is a software project that provides vector and dyadic CPU inference for large Mixture-of-Experts (MoE) models, specifically targeting 284B and 30B parameter models. It introduces two block-64 quantization formats, Fractal-64 and LUT-VQ3-64, designed for single-instruction SIMD register lookup. The project enables running models like DeepSeek-V4-Flash 284B on a pure 64GB CPU system with 256K context, requiring zero GPUs. It is licensed under Apache-2.0 and is open source.
Key features
- Fractal-64 (GGML_TYPE_Q2_0 = 42) quantization at 3.75 bpw, using 30 bytes per 64 weights, achieving 17.75 dB SQNR and 99.15% cosine similarity vs BF16.
- LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43) quantization at 3.00 bpw, using 24 bytes per 64 weights, achieving 13.57 dB SQNR and 97.78% cosine similarity vs BF16.
- Hybrid quantization combining Fractal-64 for boundary layers and residual output projections with LUT-VQ3-64 for interior expansion projections, cutting up_exps footprint by 33.3% vs Q4_0.
- Cache-aware KV management (HARP-Cache), /dev/shm Slot-0 prefix snapshots, and RouteCrack + Stockfish-PVS v2 speculative decoding.
- Sub-20ms /dev/shm Slot-0 restore for a 40-tool Claude Code prefix (2,776 tokens) in 10.4 ms on Linux.
- Cross-platform native support: Linux, macOS Metal (Apple Silicon), and Windows 11 (MinGW-w64 UCRT64).
Use cases
Running large MoE models (284B and 30B) on CPU-only systems with limited RAM (e.g., 64GB for 284B models, 16GB for 30B models). Suitable for developer laptops, Mini-PCs, desktops, and workstations. Enables agentic tool-calling, executable code generation, and long-context needle-in-a-haystack tasks with verified quality parity to BF16 baselines.
Getting started
See the repository README for installation and quick-start instructions.
License and pricing
License: Apache-2.0. Pricing: open source (not stated as paid).
Based on the project's public README and repository metadata.