shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

What is shimmy?

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

Overview of shimmy

What it is

Shimmy is a local inference server and CLI built in Rust. It uses the Airframe engine (version 0.4.5) to run GGUF models on any GPU via WebGPU. The server provides an OpenAI-API compatible endpoint for chat and text completions. It does not require Python or llama.cpp and is distributed as a single binary.

Key features

  • Pure-Rust WebGPU inference engine, runs on any GPU (Vulkan, Metal, DirectX via WebGPU).
  • OpenAI-API compatible server with model listing, JSON chat/text completions, and chat-only SSE.
  • Native GGUF model support; no Python or llama.cpp needed.
  • Single binary distribution; installable via cargo, Homebrew, Scoop, or Docker.
  • F32 KV cache and reference kernels define the validated correctness path.
  • Lazy loading by default; optional selected-model warm-up.
  • Loopback mode runs without an API key; non-loopback binds require SHIMMY_API_KEY.

Use cases

Running large language models locally on consumer GPUs for development, testing, or private inference. Suitable for users who want an OpenAI-compatible server without cloud dependencies. Can be used for chat applications, text generation, and model evaluation on local hardware.

Getting started

bashcargo install shimmy --version 2.6.6 --lockedshimmy --versionshimmy --ctx 512 --kv-quant f32 serve --model-path /path/to/model.gguf --bind 127.0.0.1:11435A working Rust toolchain and GPU driver are required. See the repository README for quickstart, model hashes, and configuration details.

License and pricing

Shimmy server code is licensed under Apache-2.0. The default dependency stack (Airframe, airframe-observe: MIT; ShimmyTok: MIT OR Apache-2.0; libfse 0.1.4: Apache-2.0 with additional grant) permits commercial use, modification, redistribution, and incorporation into products without separate commercial permission. The project is open source and free; the maintainer states it will never become a paid product. Sponsorship is accepted but not required.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/michael-a-kuykendallshimmy.svg)](https://yunodb.com/tools/michael-a-kuykendallshimmy?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/michael-a-kuykendallshimmy?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/michael-a-kuykendallshimmy.svg" alt="Featured on YunoDB" height="20"></a>