gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

What is gpustack?

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

Overview of gpustack

What it is

GPUStack is an open-source GPU cluster manager for AI model serving and GPU instance provisioning. It configures and orchestrates inference engines — vLLM, SGLang, TensorRT-LLM, or custom ones — and lets users launch SSH-accessible GPU instances on demand. The project is licensed under Apache-2.0.

Key features

  • Multi-cluster GPU management across on-premises servers, Kubernetes, and cloud providers, with GPU slicing for VRAM and compute allocation.
  • Pluggable inference engines that automatically configure vLLM, SGLang, TensorRT-LLM, and support custom engines.
  • Day 0 model support, enabling deployment of newly released models on the day they are published.
  • Performance-optimized configurations with low-latency or high-throughput modes, prefill/decode disaggregation, shared KV cache with L2 storage, and built-in speculative decoding (EAGLE3, MTP, N-grams).
  • Built-in benchmarking via Auto Tune, which explores concurrency or request rates to find peak throughput or the highest concurrency meeting SLOs.
  • On-demand SSH-accessible GPU instances for development, fine-tuning, and interactive workloads.
  • Enterprise-grade operations including automated failure recovery, load balancing, monitoring, authentication, and access control.

Use cases

GPUStack is designed for development teams, IT organizations, and service providers to deliver Model-as-a-Service at scale. It supports industry-standard APIs for LLM, voice, image, and video models. Use cases include deploying and serving AI models with high performance, provisioning GPU instances for development and fine-tuning, and managing GPU clusters across hybrid environments.

Getting started

Prerequisites: a node with at least one NVIDIA GPU (or other supported accelerators), NVIDIA driver, Docker, and NVIDIA Container Toolkit. A CPU-only node can host the GPUStack server. Only Linux is supported for worker nodes.Install GPUStack server with Docker:bashsudo docker run -d --name gpustack --restart unless-stopped -p 80:80 --volume gpustack-data:/var/lib/gpustack gpustack/gpustackRetrieve the default admin password:bashsudo docker exec gpustack cat /var/lib/gpustack/initial_admin_passwordAccess the UI at http://your_host_ip, log in with username admin and the retrieved password. Then add a cluster and worker node via the UI, and deploy a model from the Catalog. For full instructions, see the repository README.

License and pricing

GPUStack is open source under the Apache-2.0 license. Pricing is not stated; the project is listed as open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/gpustackgpustack.svg)](https://yunodb.com/tools/gpustackgpustack?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/gpustackgpustack?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/gpustackgpustack.svg" alt="Featured on YunoDB" height="20"></a>