TransformerEngine
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwel
What is TransformerEngine?
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
Overview of TransformerEngine
What it is
Transformer Engine (TE) is an NVIDIA library for accelerating Transformer model training on NVIDIA GPUs. It combines optimized building blocks and fused kernels with automatic mixed-precision-style APIs for PyTorch and JAX, so low-precision training can be adopted without rewriting a training stack. Transformer Engine manages the scaling factors, amax histories, and quantization metadata required by low-precision recipes. Its modules cover attention, linear layers, normalization, Mixture-of-Experts (MoE), and communication operations used in large-scale distributed training.
Key features
- FP8 training on NVIDIA Hopper, Ada, Blackwell, and Rubin GPUs.
- MXFP8 and NVFP4 training on NVIDIA Blackwell GPUs.
- Optimized attention, GEMM, normalization, quantization, and fused Transformer and MoE modules.
- PyTorch and JAX APIs with autocast-style contexts and configurable low-precision recipes.
- Support for tensor, sequence, context, and expert parallelism (EP), including communication overlap.
- FP16 and BF16 optimizations on NVIDIA Ampere architecture GPUs and later.
Use cases
Accelerating Transformer model training and inference on NVIDIA GPUs. Used in large-scale distributed training with Mixture-of-Experts (MoE). Integrated with popular LLM frameworks such as Hugging Face Accelerate, Lightning, NVIDIA Megatron-LM, NVIDIA NeMo Megatron Bridge, and Amazon SageMaker Model Parallel Library.
Getting started
Install the latest stable release for your framework: bash # PyTorch pip install --no-build-isolation 'transformer_engine[pytorch]' # JAX pip install --no-build-isolation 'transformer_engine[jax]' For a ready-to-run environment, use an NVIDIA NGC framework container: bash docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:-py3 docker run --gpus all -it --rm nvcr.io/nvidia/jax:-py3 Continue with the PyTorch and JAX getting started guide. For prerequisites, source builds, environment variables, and troubleshooting, see the installation guide.
License and pricing
Apache-2.0 license. Open source (free).
Based on the project's public README and repository metadata.