RustingStrata

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP specul

What is RustingStrata?

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Overview of RustingStrata

What it is

RustingStrata is a pure-Rust inference engine for Qwen GGUF models, including Qwen3 to Qwen3.8, dense and Mixture-of-Experts (MoE) architectures up to 80B+ parameters. It supports CUDA and Vulkan backends for NVIDIA and AMD GPUs, hybrid CPU+GPU expert execution, and MTP speculative decoding. The engine provides an OpenAI and Anthropic compatible server. It compiles CUDA kernels at runtime via NVRTC and caches them; no C++ build step or ggml dependency is required.

Key features

  • Pure-Rust implementation with no C++ build step.
  • Supports CUDA (NVIDIA) and Vulkan (NVIDIA, AMD) backends.
  • Hybrid CPU+GPU expert execution for MoE models when experts exceed VRAM.
  • MTP (Multi-Token Prediction) speculative decoding.
  • OpenAI and Anthropic compatible chat completions server.
  • Runtime CUDA kernel compilation via NVRTC with caching.
  • Supports a wide range of GGUF quantization types: F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, Q2_0, Q2_K–Q6_K, IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL.

Use cases

  • Running Qwen3, Qwen3.5, Qwen3.6, Qwen3.8, and Qwen3-Next models in GGUF format.
  • Local inference with GPU acceleration on NVIDIA or AMD hardware.
  • Serving an OpenAI-compatible API for chat applications, agents, or tools like Open WebUI, Jan, SillyTavern, and code editors.
  • Serving an Anthropic Messages API for Claude Code and other Anthropic-compatible clients.
  • Benchmarking and performance evaluation of Qwen models.

Getting started

shcargo build --releaseFor detailed usage instructions, including calibration, generation, chat, benchmarking, and server commands, see the repository README.

License and pricing

The license is not stated.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/rustingstudiorustingstrata.svg)](https://yunodb.com/tools/rustingstudiorustingstrata?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/rustingstudiorustingstrata?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/rustingstudiorustingstrata.svg" alt="Featured on YunoDB" height="20"></a>