gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
What is gpustack?
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Overview of gpustack
What it is
GPUStack is an open-source GPU cluster manager for AI model serving and GPU instance provisioning. It configures and orchestrates inference engines — vLLM, SGLang, TensorRT-LLM, or custom ones — and lets users launch SSH-accessible GPU instances on demand. The project is licensed under Apache-2.0.
Key features
- Multi-cluster GPU management across on-premises servers, Kubernetes, and cloud providers, with GPU slicing for VRAM and compute allocation.
- Pluggable inference engines that automatically configure vLLM, SGLang, TensorRT-LLM, and support custom engines.
- Day 0 model support, enabling deployment of newly released models on the day they are published.
- Performance-optimized configurations with low-latency or high-throughput modes, prefill/decode disaggregation, shared KV cache with L2 storage, and built-in speculative decoding (EAGLE3, MTP, N-grams).
- Built-in benchmarking via Auto Tune, which explores concurrency or request rates to find peak throughput or the highest concurrency meeting SLOs.
- On-demand SSH-accessible GPU instances for development, fine-tuning, and interactive workloads.
- Enterprise-grade operations including automated failure recovery, load balancing, monitoring, authentication, and access control.
Use cases
GPUStack is designed for development teams, IT organizations, and service providers to deliver Model-as-a-Service at scale. It supports industry-standard APIs for LLM, voice, image, and video models. Use cases include deploying and serving AI models with high performance, provisioning GPU instances for development and fine-tuning, and managing GPU clusters across hybrid environments.
Getting started
Prerequisites: a node with at least one NVIDIA GPU (or other supported accelerators), NVIDIA driver, Docker, and NVIDIA Container Toolkit. A CPU-only node can host the GPUStack server. Only Linux is supported for worker nodes.Install GPUStack server with Docker:bashsudo docker run -d --name gpustack --restart unless-stopped -p 80:80 --volume gpustack-data:/var/lib/gpustack gpustack/gpustackRetrieve the default admin password:bashsudo docker exec gpustack cat /var/lib/gpustack/initial_admin_passwordAccess the UI at http://your_host_ip, log in with username admin and the retrieved password. Then add a cluster and worker node via the UI, and deploy a model from the Catalog. For full instructions, see the repository README.
License and pricing
GPUStack is open source under the Apache-2.0 license. Pricing is not stated; the project is listed as open source.
Based on the project's public README and repository metadata.