Megatron-LM
Ongoing research training transformer models at scale
What is Megatron-LM?
Ongoing research training transformer models at scale
Overview of Megatron-LM
What it is
Megatron-LM is a GPU-optimized library for training transformer models at scale. The repository contains two components: Megatron-LM, a reference example with pre-configured training scripts ideal for research teams and quick experimentation, and Megatron Core, a composable library with GPU-optimized building blocks for custom training frameworks. It also includes Megatron Bridge for bidirectional Hugging Face ↔ Megatron checkpoint conversion.
Key features
- Advanced parallelism strategies: tensor, pipeline, data, expert, and context parallelism.
- Mixed precision support: FP16, BF16, FP8, and FP4.
- Model architectures including transformer, hybrid, and multimodal models.
- Distributed checkpoint saving, loading, and resharding.
- Inference engines and server.
- Model export (e.g., to TensorRT-LLM).
- Post-training support (quantization, distillation, pruning).
Use cases
- Research teams learning distributed training and performing quick experimentation.
- Framework developers and ML engineers building custom training pipelines.
- Reproducing training runs and validating checkpoint resume at scale.
Getting started
Python 3.12+ and PyTorch 2.6.0+ are recommended.
Install from PyPI:
uv pip install megatron-core
Or clone and install from source:
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
uv pip install -e .
If the build runs out of memory, limit parallel compilation by setting MAX_JOBS (e.g., MAX_JOBS=4 uv pip install -e .).
License and pricing
The license is not stated (NOASSERTION). The software is open source and free.
Based on the project's public README and repository metadata.