NextTransformer
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonom
What is NextTransformer?
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
Overview of NextTransformer
What it is
NextTransformer is an open-source framework for next-generation Transformer architectures. It provides compute-matched benchmarking and a platform for autonomous architecture research. The framework trains various architectures on the same data, for the same 49.152B tokens, at the same theoretical training FLOPs per token, and evaluates them at the same checkpoints. It includes an observatory with learning curves, leaderboard, tasks, architecture diagrams, and model cards.
Key features
- Compute-matched benchmarking: all methods trained under identical budget (49.152B tokens, 4.672 GFLOPs per token) and evaluated at same checkpoints.
- Model zoo: includes families such as Dense Baseline, Residual, Recurrent, Sparse, Linear, Hybrid, and Attention Variants.
- Auto-research arena: allows any LLM to propose and train architectures under the same protocol, with Arena v2 planned for November 14, 2026.
- Open-source with MIT license, 31 finished runs and 36 model cards as of October 31, 2026.
- Community projects: NextTransformer-V (multimodal) and ResearchBoost-Agent (training optimization).
Use cases
- Research on next-generation Transformer architectures with fair comparison.
- Benchmarking new attention mechanisms, recurrent depth, sparse and linear attention, and hybrid models.
- Autonomous architecture search via the arena.
- Community contributions and extensions for multimodal and optimization research.
Getting started
Install on Linux with NVIDIA GPUs using uv:
git clone https://github.com/Interactive-Active-AI/NextTransformer.git && cd NextTransformer
uv sync --frozen --extra gpu --extra upstream && source .venv/bin/activate
uv pip install --no-deps --require-hashes -r requirements/linear.txt
Prepare data (ClimbMix ~600 GB) and train on one node with 8 GPUs:
launch/train.sh --config configs/baseline/dense_full.json --site-config site.local.json \
--output runs/dense_full --micro-batch 8 --formal --resume auto
See the repository README for multi-node training and evaluation details.
License and pricing
Licensed under MIT. Open-source, no pricing information stated.
Based on the project's public README and repository metadata.