llama-modchip

Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).

What is llama-modchip?

Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).

Overview of llama-modchip

What it is

llama-modchip is a software project that provides vector and dyadic CPU inference for large Mixture-of-Experts (MoE) models, specifically targeting 284B and 30B parameter models. It introduces two block-64 quantization formats, Fractal-64 and LUT-VQ3-64, designed for single-instruction SIMD register lookup. The project enables running models like DeepSeek-V4-Flash 284B on a pure 64GB CPU system with 256K context, requiring zero GPUs. It is licensed under Apache-2.0 and is open source.

Key features

  • Fractal-64 (GGML_TYPE_Q2_0 = 42) quantization at 3.75 bpw, using 30 bytes per 64 weights, achieving 17.75 dB SQNR and 99.15% cosine similarity vs BF16.
  • LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43) quantization at 3.00 bpw, using 24 bytes per 64 weights, achieving 13.57 dB SQNR and 97.78% cosine similarity vs BF16.
  • Hybrid quantization combining Fractal-64 for boundary layers and residual output projections with LUT-VQ3-64 for interior expansion projections, cutting up_exps footprint by 33.3% vs Q4_0.
  • Cache-aware KV management (HARP-Cache), /dev/shm Slot-0 prefix snapshots, and RouteCrack + Stockfish-PVS v2 speculative decoding.
  • Sub-20ms /dev/shm Slot-0 restore for a 40-tool Claude Code prefix (2,776 tokens) in 10.4 ms on Linux.
  • Cross-platform native support: Linux, macOS Metal (Apple Silicon), and Windows 11 (MinGW-w64 UCRT64).

Use cases

Running large MoE models (284B and 30B) on CPU-only systems with limited RAM (e.g., 64GB for 284B models, 16GB for 30B models). Suitable for developer laptops, Mini-PCs, desktops, and workstations. Enables agentic tool-calling, executable code generation, and long-context needle-in-a-haystack tasks with verified quality parity to BF16 baselines.

Getting started

See the repository README for installation and quick-start instructions.

License and pricing

License: Apache-2.0. Pricing: open source (not stated as paid).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/synchopatellama-modchip.svg)](https://yunodb.com/tools/synchopatellama-modchip?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/synchopatellama-modchip?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/synchopatellama-modchip.svg" alt="Featured on YunoDB" height="20"></a>