distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
What is distributed-llama?
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Overview of distributed-llama
What it is
Distributed Llama is an open-source project that enables distributed LLM inference by connecting multiple home devices into a cluster. It uses tensor parallelism and high-speed Ethernet synchronization to accelerate inference performance. The project supports Linux, macOS, and Windows, and is optimized for ARM and x86_64 AVX2 CPUs.
Key features
- Supports distributed inference across 1, 2, 4... 2^n nodes, with the maximum number of nodes equal to the number of KV heads in the model.
- Provides a single-command setup for root nodes using
python launch.pywith pre-configured model commands (e.g., Llama 3.1 8B, Llama 3.3 70B, Qwen 3 series). - Includes CLI commands:
dllama inference,dllama chat,dllama worker, anddllama-api. - Supports quantizations:
q40model withq80buffer-float-type, andf32model withf32buffer-float-type. - Allows manual conversion of Hugging Face models.
Use cases
- Running large language models on a cluster of home devices (e.g., Raspberry Pi, Mac Mini, PCs) to achieve faster inference than a single device.
- Experimenting with distributed inference for research or hobbyist projects.
- Deploying LLM inference in environments where dedicated GPU hardware is not available but multiple CPU-based devices are.
Getting started
Python 3 and a C++ compiler are required. To set up a root node with a specific model, run one of the following commands:python launch.py llama3_1_8b_instruct_q40python launch.py llama3_3_70b_instruct_q40python launch.py qwen3_8b_q40For detailed instructions on running on Linux, macOS, Windows, Raspberry Pi, or GPU, see the repository README.
License and pricing
This project is released under the MIT license. It is open-source and free to use.
Based on the project's public README and repository metadata.