paddler

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale πŸ“πŸ¦™ Alternative to projects like llm-d, Docker Model Runner, e

What is paddler?

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale πŸ“πŸ¦™ Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

Overview of paddler

What it is

Paddler is an open-source LLM/VLM load balancer and serving platform for self-hosting large language models (and vision-language models) at scale. It is designed as an alternative to projects like llm-d or Docker Model Runner, with fewer moving parts and simpler deployments built around the ggml ecosystem. Paddler runs on both CPU and GPU.

Key features

  • Inference through a built-in llama.cpp engine - LLM-specific load balancing - Works through agents that can be added dynamically, allowing integration with autoscaling tools - Request buffering, enabling scaling from zero hosts - Dynamic model swapping - Built-in web admin panel for management, monitoring, and testing - Observability metrics

Use cases

Paddler is intended for product teams that need LLM inference and embeddings in their features, DevOps/LLMOps teams that need to run and deploy LLMs at scale, organizations handling sensitive data with high compliance and privacy requirements (e.g., medical, financial), organizations wanting to achieve predictable LLM costs instead of per-token pricing, and product leaders who need reliable model performance for consistent user experience.

Getting started

Paddler is self-contained in a single binary file. Obtain the paddler binary by downloading the latest release from GitHub releases or building from source (MSRV is 1.88.0). To start the balancer, run: sh paddler balancer --inference-addr 127.0.0.1:8061 --management-addr 127.0.0.1:8060 --web-admin-panel-addr 127.0.0.1:8062 To start an agent with 4 slots, run: sh paddler agent --management-addr 127.0.0.1:8060 --slots 4

License and pricing

The license is Apache-2.0. The project is open source; no pricing information is stated in the provided metadata or README.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/intenteepaddler.svg)](https://yunodb.com/tools/intenteepaddler?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/intenteepaddler?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/intenteepaddler.svg" alt="Featured on YunoDB" height="20"></a>