rl

A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.

What is rl?

A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.

Overview of rl

What it is

TorchRL is a PyTorch-native toolkit for reinforcement learning, decision making, robotics, and simulation. It is a collection of composable pieces for building RL systems while keeping the code close to the PyTorch programming model. The library is built around three ideas: data should have names, structure, batch dimensions, and devices all the way through the training loop; environments, policies, replay buffers, objectives, and collectors should be independent modules that can be swapped without rewriting the rest of the stack; research code should scale from a local prototype to vectorized, multiprocess, distributed, compiled, recurrent, multi-agent, model-based, or offline workflows without changing the data model. That common data model is TensorDict, a dictionary-like tensor container with PyTorch operations, device transfers, shared-memory support, memmaps, lazy views, and nn.Module wrappers.

Key features

  • TensorDict-first pipelines: RL data is represented as a TensorDict that preserves named fields, batch dimensions, and devices throughout the training loop.
  • Environments and transforms: Includes native environments, wrappers for Gymnasium, DM Control, Brax, and others, plus environment transforms for observation normalization, image conversion, reward transforms, action masking, and more.
  • Collectors and execution models: Single-process, async, multiprocess, and distributed collectors that bridge policies and environments, batching trajectories and handling devices.
  • Replay buffers: Store, sample, prioritize, and transform trajectory data, with support for writing complete episodes via replay_write_mode='trajectory'.
  • Loss and value-estimation components: Reusable loss modules and value-estimation components for RL training.
  • Robotics and deployment: MuJoCo Menagerie and MicroDuck environments, low-level controller composition, and TensorDict policy export to ONNX.
  • Model-based and offline RL: TD-MPC2, Flow Q-Learning, and Transitive RL implementations.

Use cases

TorchRL is designed for reinforcement learning, decision making, robotics, and simulation. It supports recurrent RL, MuJoCo-based control, multi-agent training, replay-buffer and collector infrastructure, and reusable loss/value-estimation components. It can be used for model-based RL, offline RL, robot learning, and training at scale with vectorized, multiprocess, distributed, compiled, or offline workflows.

Getting started

pythonimport torchfrom tensordict.nn import TensorDictModulefrom torch import nnfrom torchrl.envs import PendulumEnv, StepCounter, TransformedEnvenv = TransformedEnv(PendulumEnv(), StepCounter(max_steps=200))policy = TensorDictModule( nn.Sequential( nn.LazyLinear(64), nn.Tanh(), nn.Linear(64, 1), nn.Tanh(), ), in_keys=[observation], out_keys=[action],)rollout = env.rollout(max_steps=32, policy=policy)

License and pricing

TorchRL is open source under the MIT license.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/pytorchrl.svg)](https://yunodb.com/tools/pytorchrl?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/pytorchrl?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/pytorchrl.svg" alt="Featured on YunoDB" height="20"></a>