tokenspeed
TokenSpeed is a speed-of-light LLM inference engine.
What is tokenspeed?
TokenSpeed is a speed-of-light LLM inference engine.
Overview of tokenspeed
What it is
TokenSpeed is an open-source LLM inference engine designed for agentic workloads. It is developed by the non-profit LightSeek Foundation and is part of the PyTorch Ecosystem Landscape. The engine separates the control plane from the execution plane: the control plane is implemented in C++ as a finite-state machine, using the type system to enforce safe resource management at compile time, while the execution plane is implemented in Python for fast iteration. TokenSpeed treats kernels as a modular subsystem with a portable public API and a centralized registry. It supports heterogeneous accelerators through an extensible plugin mechanism.
Key features
- Control plane in C++ as a finite-state machine with compile-time safety for request lifecycles and KV cache state.
- Execution plane in Python, enabling rapid development and lower cognitive load.
- Pluggable, layered kernel system with a portable public API and centralized registry, including an MLA (Multi-head Latent Attention) implementation for Blackwell.
- Entrypoint using SMG and Dynamo from Rust over msgpack wire, with no Python on the request path.
- Backed by the non-profit LightSeek Foundation with hardware and engineering support from vendors such as NVIDIA and AMD.
- Supports agentic workloads with performance comparable to TensorRT-LLM and usability similar to vLLM.
Use cases
TokenSpeed is intended for production agentic workloads, such as serving large language models in multi-turn agent scenarios. It can be used to deploy models like Kimi K3, Qwen3.8-Flash-Next, and others on NVIDIA and AMD GPUs. The engine is suitable for scenarios requiring high throughput and low latency, such as real-time AI agents and interactive applications.
Getting started
See the repository README for installation and quick-start instructions.
License and pricing
TokenSpeed is open source under the MIT license. Pricing is not stated; the project is listed as open source.
Based on the project's public README and repository metadata.