mesh-llm
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
What is mesh-llm?
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
Overview of mesh-llm
What it is
Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. It allows starting one node and adding more later, with the mesh deciding whether a model runs locally, routes to a peer, or uses Skippy stage splits for models too large for one box.
Key features
- Single-machine fit first: if one node can host the full model, it serves locally without stage traffic.
- Mesh routing: every node exposes the same
/v1API; requests are routed by themodelfield to the peer that can serve that model. - Encrypted peer transport: QUIC end-to-end encrypts traffic between Mesh nodes; Iroh relays forward encrypted packets without reading payloads.
- Owner-control plane: operator config and inventory actions use an additive
mesh-llm-control/2lane with explicit endpoint bootstrap. - Skippy stage splits: large dense models can load as package-backed layer stages; the coordinator plans contiguous layer ranges.
- Layer packages: package repositories contain
model-package.jsonplus GGUF fragments so peers fetch only needed pieces. - Public discovery: published meshes advertise through Nostr discovery; private meshes stay invite-token based.
Use cases
- Running large language models that do not fit on a single machine by splitting across multiple nodes.
- Creating private or public meshes for collaborative AI inference.
- Integrating with agent tools like Goose, OpenCode, Claude Code, or Pi via
mesh-llmsubcommands. - Debugging and development with local model-only serving (bypassing mesh networking).
Getting started
Install the latest release executable:
curl -fsSL https://meshllm.cloud/install.sh | bash
On Windows, use PowerShell:
irm https://meshllm.cloud/install.ps1 | iex
Install the Apple Silicon Homebrew formula with brew install Mesh-LLM/tap/mesh-llm. Finish setup:
mesh-llm setup
Join the public mesh and start serving:
mesh-llm serve --auto
License and pricing
Licensed under Apache-2.0. The project is open source (pricing_type: open_source). No pricing information is stated beyond the open source license.
Based on the project's public README and repository metadata.