tokenspeed

TokenSpeed is a speed-of-light LLM inference engine.

What is tokenspeed?

TokenSpeed is a speed-of-light LLM inference engine.

Overview of tokenspeed

What it is

TokenSpeed is an open-source LLM inference engine designed for agentic workloads. It is developed by the non-profit LightSeek Foundation and is part of the PyTorch Ecosystem Landscape. The engine separates the control plane from the execution plane: the control plane is implemented in C++ as a finite-state machine, using the type system to enforce safe resource management at compile time, while the execution plane is implemented in Python for fast iteration. TokenSpeed treats kernels as a modular subsystem with a portable public API and a centralized registry. It supports heterogeneous accelerators through an extensible plugin mechanism.

Key features

  • Control plane in C++ as a finite-state machine with compile-time safety for request lifecycles and KV cache state.
  • Execution plane in Python, enabling rapid development and lower cognitive load.
  • Pluggable, layered kernel system with a portable public API and centralized registry, including an MLA (Multi-head Latent Attention) implementation for Blackwell.
  • Entrypoint using SMG and Dynamo from Rust over msgpack wire, with no Python on the request path.
  • Backed by the non-profit LightSeek Foundation with hardware and engineering support from vendors such as NVIDIA and AMD.
  • Supports agentic workloads with performance comparable to TensorRT-LLM and usability similar to vLLM.

Use cases

TokenSpeed is intended for production agentic workloads, such as serving large language models in multi-turn agent scenarios. It can be used to deploy models like Kimi K3, Qwen3.8-Flash-Next, and others on NVIDIA and AMD GPUs. The engine is suitable for scenarios requiring high throughput and low latency, such as real-time AI agents and interactive applications.

Getting started

See the repository README for installation and quick-start instructions.

License and pricing

TokenSpeed is open source under the MIT license. Pricing is not stated; the project is listed as open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/lightseekorgtokenspeed.svg)](https://yunodb.com/tools/lightseekorgtokenspeed?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/lightseekorgtokenspeed?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/lightseekorgtokenspeed.svg" alt="Featured on YunoDB" height="20"></a>