kyojin

Kyojin: open-source engine that runs 100 GB-class open models (GLM-5.3, Qwen3.8, DeepSeek V4.1) on one desktop machine. AMD Strix Halo, NVIDIA RTX and Apple Mac

What is kyojin?

Kyojin: open-source engine that runs 100 GB-class open models (GLM-5.3, Qwen3.8, DeepSeek V4.1) on one desktop machine. AMD Strix Halo, NVIDIA RTX and Apple Mac. OpenAI API, built on ExLlamaV3.

Overview of kyojin

What it is

Kyojin is an open-source inference engine designed to run large open models on a single mini PC or graphics card. It is built on ExLlamaV3 and supports AMD (ROCm) and NVIDIA (CUDA) hardware. The engine claims to be about 2× faster than llama.cpp for certain workloads.

Key features

  • Speculative decoding for improved token generation speed.
  • Support for long contexts up to 500K tokens.
  • Uncensored mode for certain models (GLM-5.3-Flash, MiMo-V2.6-Flash, Qwen3.8-Flash-Next) that reduces refusal rates without modifying weights.
  • Experimental support for DeepSeek-V4.1-Flash (763B parameters) on a single 128 GB machine with NVMe SSD tiering.
  • Quantized KV cache (6/6 bits for Flint, 4/4 bits for Qwen27) to fit larger contexts on limited VRAM.
  • Prometheus /metrics endpoint for monitoring.

Use cases

  • Running large language models (100 GB-class) on consumer hardware with 128 GB unified memory.
  • Long-context document analysis (up to 500K tokens).
  • Local inference with OpenAI-compatible API for development and testing.
  • Experimental deployment of very large models (e.g., 763B) on a single machine with SSD offloading.

Getting started

bashgit clone https://github.com/Yamz-Labs/kyojin && cd kyojinhf download yamz-labs/Qwen3.8-Flash-Next-EXL3-Yamz --local-dir ./qwen-packpython tools/qwen/serve.py --model ./qwen-pack --port 8000 -c 131072Install ROCm torch and build once before running (see doc/install.md).

License and pricing

MIT license. The project is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/yamz-labskyojin.svg)](https://yunodb.com/tools/yamz-labskyojin?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/yamz-labskyojin?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/yamz-labskyojin.svg" alt="Featured on YunoDB" height="20"></a>