AgileRL
The complete post-training stack for LLMs, plus RL for any problem. SFT, DPO and multi-turn agentic RL, with evolutionary hyperparameter optimization built in.
What is AgileRL?
The complete post-training stack for LLMs, plus RL for any problem. SFT, DPO and multi-turn agentic RL, with evolutionary hyperparameter optimization built in.
Overview of AgileRL
What it is
AgileRL is an open-source library that provides a complete post-training stack for large language models (LLMs) and classic reinforcement learning (RL) algorithms. It includes evolutionary hyperparameter optimization (HPO) built in, and supports training from supervised fine-tuning through preference tuning to multi-turn agentic RL. The library also covers classic deep RL for single-agent, multi-agent, offline, and contextual bandit problems. It is licensed under Apache-2.0 and is available on GitHub.
Key features
- Full LLM post-training pipeline: SFT, DPO, and RL with GRPO, CISPO, GSPO, REINFORCE, or PPO.
- Multi-turn agentic RL: train on any OpenEnv environment, a dataset plus reward function, or a remote env server.
- LoRA and QLoRA support for memory-efficient fine-tuning.
- Torch-native parallelism with data parallel or FSDP2 sharding, launched via
torchrun. - Memory optimizations including chunked fused log-probs, activation checkpointing and offload, CPU offload for optimizer state or parameters, and trainer offload during rollout.
- Model-specific optimizations for dense, MoE, and hybrid Mamba models (e.g., MoE LoRA per expert, fused kernels for Nemotron-H).
- Fast rollouts using vLLM generation, with optional colocation of trainer and vLLM on a single GPU.
- Evolutionary HPO for LLMs: hyperparameters like learning rate and KL penalty tune themselves during the run.
- Agent-friendly CLI: training runs defined by a single YAML manifest and one command.
- Async training at scale on Arena (managed GPU clusters) with models up to 120B parameters.
- Deploy and chat: deploy the best checkpoint on Arena and interact via CLI.
- Classic RL algorithms (on-policy, off-policy, offline, multi-agent, contextual bandits) all with evolutionary HPO.
Use cases
- Post-training of open-weights LLMs: from SFT and DPO to multi-turn agentic RL.
- Training and deploying RL agents for single-agent tasks (e.g., Gymnasium environments), multi-agent tasks (e.g., PettingZoo), offline RL, and contextual bandits.
- Scaling LLM post-training to large models (up to 120B) on managed GPU clusters via Arena.
- Rapid hyperparameter tuning without separate trial runs, using population-based evolutionary HPO.
Getting started
bash pip install agilerl # classic RL + Arena CLI pip install agilerl[llm] # LLM post-training For development tip of main branch: pip install git+https://github.com/AgileRL/AgileRL.git@main. For development mode: clone the repo and run pip install -e ..
License and pricing
License: Apache-2.0. Pricing: open source (no stated cost).
Based on the project's public README and repository metadata.