TensorRT
PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT
What is TensorRT?
PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT
Overview of TensorRT
What it is
Torch-TensorRT is a compiler for PyTorch, TorchScript, and TorchFX that targets NVIDIA GPUs using TensorRT. It is designed to improve inference performance for PyTorch models on NVIDIA platforms. The project is open source under the BSD-3-Clause license and is hosted on GitHub at pytorch/TensorRT. It can be installed via PyPI or used through the NVIDIA NGC PyTorch Container. The software supports Linux AMD64, Linux SBSA, Windows (Dynamo only), and Linux Jetson platforms. Dependencies include Bazel 8.1.1, Libtorch 2.16.0.dev, CUDA 13.4, and TensorRT 11.4.0.106. The project follows a deprecation policy with a 6-month migration period after deprecation notices are issued.
Key features
- Accelerates inference latency by up to 5x compared to eager execution in one line of code.
- Provides a
torch.compilebackend namedtensorrtfor just-in-time compilation. - Offers an export-style workflow for ahead-of-time optimization and serialization, deployable in PyTorch or C++ (libtorch).
- Supports FP8 inference and optimization of models like Stable Diffusion and Hugging Face LLMs.
- Includes tools to resolve graph breaks and boost performance (coming soon).
Use cases
- Optimizing any PyTorch model for faster inference on NVIDIA GPUs.
- Deploying optimized models in production environments using either Python or C++ runtimes.
- Accelerating diffusion models, large language models, and other deep learning workloads.
- Running models in FP8 precision for improved performance on compatible hardware.
Getting started
Stable versions can be installed via PyPI: bash pip install torch-tensorrt Nightly versions are available from the PyTorch package index: bash pip install --pre torch-tensorrt --index-url https://download.pytorch.org/whl/nightly/cu134 --extra-index-url https://pypi.org/simple For quick usage, use torch.compile with the tensorrt backend: python import torch import torch_tensorrt model = MyModel().eval().cuda() x = torch.randn((1, 3, 224, 224)).cuda() optimized_model = torch.compile(model, backend='tensorrt') optimized_model(x) For export-style optimization, use torch_tensorrt.compile and torch_tensorrt.save. See the repository README for more details.
License and pricing
The project is licensed under BSD-3-Clause. It is open source and free to use. Pricing is not stated; the license is the only cost-related information provided.
Based on the project's public README and repository metadata.