xtuner
A Next-Generation Training Engine Built for Ultra-Large MoE Models
What is xtuner?
A Next-Generation Training Engine Built for Ultra-Large MoE Models
Overview of xtuner
What it is
XTuner V1 is a next-generation LLM training engine specifically designed for ultra-large-scale Mixture-of-Experts (MoE) models. It is optimized for the mainstream MoE training scenarios prevalent in today's academic research, moving beyond traditional 3D parallel training architectures. The project is hosted on GitHub under the InternLM organization and is released under the Apache License 2.0.
Key features
- Dropless Training: Trains 200B-scale MoE models without expert parallelism; 600B models require only intra-node expert parallelism. The optimized parallelism strategy uses a smaller expert parallelism dimension compared to traditional 3D approaches, enabling more efficient dropless training.
- Long Sequence Support: Trains 200B MoE models on 64k sequence lengths without sequence parallelism through advanced memory optimization techniques. Full support for DeepSpeed Ulysses sequence parallelism with linearly scalable maximum sequence length. Maintains stability despite expert load imbalance during long sequence training.
- Superior Efficiency: Supports MoE training up to 1T parameters. First to achieve FSDP training throughput that surpasses traditional 3D parallel schemes for MoE models above 200B scale. Achieves training efficiency on Ascend A3 Supernode that exceeds NVIDIA H800.
Use cases
XTuner V1 is intended for pre-training, instruction fine-tuning, and reinforcement learning of ultra-large MoE models. It supports multimodal pre-training and supervised fine-tuning for vision-language models, as well as GRPO (Group Relative Policy Optimization). The roadmap includes future support for MPO, DAPO, and multi-turn agentic RL. The engine is designed to integrate with inference frameworks such as LMDeploy, vLLM, and SGLang.
Getting started
See the repository README for installation and quick-start instructions.
License and pricing
This project is released under the Apache License 2.0. The pricing type is open source.
Based on the project's public README and repository metadata.