kyojin
Kyojin: open-source engine that runs 100 GB-class open models (GLM-5.3, Qwen3.8, DeepSeek V4.1) on one desktop machine. AMD Strix Halo, NVIDIA RTX and Apple Mac
What is kyojin?
Kyojin: open-source engine that runs 100 GB-class open models (GLM-5.3, Qwen3.8, DeepSeek V4.1) on one desktop machine. AMD Strix Halo, NVIDIA RTX and Apple Mac. OpenAI API, built on ExLlamaV3.
Overview of kyojin
What it is
Kyojin is an open-source inference engine designed to run large open models on a single mini PC or graphics card. It is built on ExLlamaV3 and supports AMD (ROCm) and NVIDIA (CUDA) hardware. The engine claims to be about 2× faster than llama.cpp for certain workloads.
Key features
- Speculative decoding for improved token generation speed.
- Support for long contexts up to 500K tokens.
- Uncensored mode for certain models (GLM-5.3-Flash, MiMo-V2.6-Flash, Qwen3.8-Flash-Next) that reduces refusal rates without modifying weights.
- Experimental support for DeepSeek-V4.1-Flash (763B parameters) on a single 128 GB machine with NVMe SSD tiering.
- Quantized KV cache (6/6 bits for Flint, 4/4 bits for Qwen27) to fit larger contexts on limited VRAM.
- Prometheus
/metricsendpoint for monitoring.
Use cases
- Running large language models (100 GB-class) on consumer hardware with 128 GB unified memory.
- Long-context document analysis (up to 500K tokens).
- Local inference with OpenAI-compatible API for development and testing.
- Experimental deployment of very large models (e.g., 763B) on a single machine with SSD offloading.
Getting started
bashgit clone https://github.com/Yamz-Labs/kyojin && cd kyojinhf download yamz-labs/Qwen3.8-Flash-Next-EXL3-Yamz --local-dir ./qwen-packpython tools/qwen/serve.py --model ./qwen-pack --port 8000 -c 131072Install ROCm torch and build once before running (see doc/install.md).
License and pricing
MIT license. The project is open source.
Based on the project's public README and repository metadata.