shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
What is shimmy?
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Overview of shimmy
What it is
Shimmy is a local inference server and CLI built in Rust. It uses the Airframe engine (version 0.4.5) to run GGUF models on any GPU via WebGPU. The server provides an OpenAI-API compatible endpoint for chat and text completions. It does not require Python or llama.cpp and is distributed as a single binary.
Key features
- Pure-Rust WebGPU inference engine, runs on any GPU (Vulkan, Metal, DirectX via WebGPU).
- OpenAI-API compatible server with model listing, JSON chat/text completions, and chat-only SSE.
- Native GGUF model support; no Python or llama.cpp needed.
- Single binary distribution; installable via cargo, Homebrew, Scoop, or Docker.
- F32 KV cache and reference kernels define the validated correctness path.
- Lazy loading by default; optional selected-model warm-up.
- Loopback mode runs without an API key; non-loopback binds require SHIMMY_API_KEY.
Use cases
Running large language models locally on consumer GPUs for development, testing, or private inference. Suitable for users who want an OpenAI-compatible server without cloud dependencies. Can be used for chat applications, text generation, and model evaluation on local hardware.
Getting started
bashcargo install shimmy --version 2.6.6 --lockedshimmy --versionshimmy --ctx 512 --kv-quant f32 serve --model-path /path/to/model.gguf --bind 127.0.0.1:11435A working Rust toolchain and GPU driver are required. See the repository README for quickstart, model hashes, and configuration details.
License and pricing
Shimmy server code is licensed under Apache-2.0. The default dependency stack (Airframe, airframe-observe: MIT; ShimmyTok: MIT OR Apache-2.0; libfse 0.1.4: Apache-2.0 with additional grant) permits commercial use, modification, redistribution, and incorporation into products without separate commercial permission. The project is open source and free; the maintainer states it will never become a paid product. Sponsorship is accepted but not required.
Based on the project's public README and repository metadata.