distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

What is distributed-llama?

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

Overview of distributed-llama

What it is

Distributed Llama is an open-source project that enables distributed LLM inference by connecting multiple home devices into a cluster. It uses tensor parallelism and high-speed Ethernet synchronization to accelerate inference performance. The project supports Linux, macOS, and Windows, and is optimized for ARM and x86_64 AVX2 CPUs.

Key features

  • Supports distributed inference across 1, 2, 4... 2^n nodes, with the maximum number of nodes equal to the number of KV heads in the model.
  • Provides a single-command setup for root nodes using python launch.py with pre-configured model commands (e.g., Llama 3.1 8B, Llama 3.3 70B, Qwen 3 series).
  • Includes CLI commands: dllama inference, dllama chat, dllama worker, and dllama-api.
  • Supports quantizations: q40 model with q80 buffer-float-type, and f32 model with f32 buffer-float-type.
  • Allows manual conversion of Hugging Face models.

Use cases

  • Running large language models on a cluster of home devices (e.g., Raspberry Pi, Mac Mini, PCs) to achieve faster inference than a single device.
  • Experimenting with distributed inference for research or hobbyist projects.
  • Deploying LLM inference in environments where dedicated GPU hardware is not available but multiple CPU-based devices are.

Getting started

Python 3 and a C++ compiler are required. To set up a root node with a specific model, run one of the following commands:python launch.py llama3_1_8b_instruct_q40python launch.py llama3_3_70b_instruct_q40python launch.py qwen3_8b_q40For detailed instructions on running on Linux, macOS, Windows, Raspberry Pi, or GPU, see the repository README.

License and pricing

This project is released under the MIT license. It is open-source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/b4rtazdistributed-llama.svg)](https://yunodb.com/tools/b4rtazdistributed-llama?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/b4rtazdistributed-llama?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/b4rtazdistributed-llama.svg" alt="Featured on YunoDB" height="20"></a>