halogen-flash-server

Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

What is halogen-flash-server?

Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

Overview of halogen-flash-server

What it is

halogen-flash-server is a local inference server that runs Qwen3.8-Flash-Next on AMD Strix Halo hardware. It is designed for agent use cases, providing fast response times. It exposes both OpenAI-compatible and Anthropic-compatible APIs on a single port. The server also supports small NPU models.

Key features

  • Runs Qwen3.8-Flash-Next on AMD Strix Halo with custom kernels written specifically for this GPU and model family.
  • Decode speeds: up to 94 tok/s on repetitive output, 73 tok/s writing code, 54 tok/s prose (measured on 0.18.0 default configuration).
  • Prefill speeds: 1,821 tok/s for 8,192-token prompt, 1,740 tok/s for 32,768 tokens, 1,675 tok/s for 131,072 tokens.
  • Agent turn latency: first token in about 1.6 seconds for a long session (30,000 to 96,000 tokens context).
  • Supports speculative decoding with two draft sources (model's own draft head and prompt lookup), verified to produce byte-identical output at temperature 0.
  • Can load llama.cpp GGUF files of the same model directly without conversion.
  • Provides both OpenAI and Anthropic API endpoints on one port.

Use cases

  • Running local inference for AI agents that require fast turn-around times.
  • Coding agents that need to process long conversation histories.
  • Local development and testing with frontier-like API behavior.
  • Deploying on AMD Strix Halo hardware for private inference.

Getting started

bash mkdir -p ~/halogen-models podman run --rm -p 8731:8731 --device /dev/kfd --device /dev/dri --group-add keep-groups --ipc=host --ulimit memlock=-1:-1 -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next -v ~/halogen-models:/models ghcr.io/peonist-ai/halogen-flash-server:0.18.17 One condition: set the BIOS carve-out (UMA frame buffer) to its minimum.

License and pricing

The engine is open source. The license is not explicitly stated in the README; the engine is distributed under the terms in LICENSE.md. Model weights are licensed separately by their original authors.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/peonist-aihalogen-flash-server.svg)](https://yunodb.com/tools/peonist-aihalogen-flash-server?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/peonist-aihalogen-flash-server?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/peonist-aihalogen-flash-server.svg" alt="Featured on YunoDB" height="20"></a>