halogen-flash-server
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
What is halogen-flash-server?
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
Overview of halogen-flash-server
What it is
halogen-flash-server is a local inference server that runs Qwen3.8-Flash-Next on AMD Strix Halo hardware. It is designed for agent use cases, providing fast response times. It exposes both OpenAI-compatible and Anthropic-compatible APIs on a single port. The server also supports small NPU models.
Key features
- Runs Qwen3.8-Flash-Next on AMD Strix Halo with custom kernels written specifically for this GPU and model family.
- Decode speeds: up to 94 tok/s on repetitive output, 73 tok/s writing code, 54 tok/s prose (measured on 0.18.0 default configuration).
- Prefill speeds: 1,821 tok/s for 8,192-token prompt, 1,740 tok/s for 32,768 tokens, 1,675 tok/s for 131,072 tokens.
- Agent turn latency: first token in about 1.6 seconds for a long session (30,000 to 96,000 tokens context).
- Supports speculative decoding with two draft sources (model's own draft head and prompt lookup), verified to produce byte-identical output at temperature 0.
- Can load llama.cpp GGUF files of the same model directly without conversion.
- Provides both OpenAI and Anthropic API endpoints on one port.
Use cases
- Running local inference for AI agents that require fast turn-around times.
- Coding agents that need to process long conversation histories.
- Local development and testing with frontier-like API behavior.
- Deploying on AMD Strix Halo hardware for private inference.
Getting started
bash mkdir -p ~/halogen-models podman run --rm -p 8731:8731 --device /dev/kfd --device /dev/dri --group-add keep-groups --ipc=host --ulimit memlock=-1:-1 -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next -v ~/halogen-models:/models ghcr.io/peonist-ai/halogen-flash-server:0.18.17 One condition: set the BIOS carve-out (UMA frame buffer) to its minimum.
License and pricing
The engine is open source. The license is not explicitly stated in the README; the engine is distributed under the terms in LICENSE.md. Model weights are licensed separately by their original authors.
Based on the project's public README and repository metadata.