Megatron-LM

Ongoing research training transformer models at scale

What is Megatron-LM?

Ongoing research training transformer models at scale

Overview of Megatron-LM

What it is

Megatron-LM is a GPU-optimized library for training transformer models at scale. The repository contains two components: Megatron-LM, a reference example with pre-configured training scripts ideal for research teams and quick experimentation, and Megatron Core, a composable library with GPU-optimized building blocks for custom training frameworks. It also includes Megatron Bridge for bidirectional Hugging Face ↔ Megatron checkpoint conversion.

Key features

  • Advanced parallelism strategies: tensor, pipeline, data, expert, and context parallelism.
  • Mixed precision support: FP16, BF16, FP8, and FP4.
  • Model architectures including transformer, hybrid, and multimodal models.
  • Distributed checkpoint saving, loading, and resharding.
  • Inference engines and server.
  • Model export (e.g., to TensorRT-LLM).
  • Post-training support (quantization, distillation, pruning).

Use cases

  • Research teams learning distributed training and performing quick experimentation.
  • Framework developers and ML engineers building custom training pipelines.
  • Reproducing training runs and validating checkpoint resume at scale.

Getting started

Python 3.12+ and PyTorch 2.6.0+ are recommended.

Install from PyPI:

uv pip install megatron-core

Or clone and install from source:

git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
uv pip install -e .

If the build runs out of memory, limit parallel compilation by setting MAX_JOBS (e.g., MAX_JOBS=4 uv pip install -e .).

License and pricing

The license is not stated (NOASSERTION). The software is open source and free.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/nvidiamegatron-lm.svg)](https://yunodb.com/tools/nvidiamegatron-lm?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/nvidiamegatron-lm?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/nvidiamegatron-lm.svg" alt="Featured on YunoDB" height="20"></a>