verl-agent

Overview of verl-agent

What it is

verl-agent is an extension of veRL, specifically designed for training large language model (LLM) agents via reinforcement learning (RL). It is the official code for the paper “Group-in-Group Policy Optimization for LLM Agent Training” (GiGPO), accepted at NeurIPS 2025. The framework proposes a step-independent multi-turn rollout mechanism that allows fully customizable per-step input structures, history management, and memory modules, making it scalable for long-horizon, multi-turn RL training (e.g., tasks in ALFWorld requiring up to 50 steps).

Key features

• Multi-turn agent-environment interaction with step-wise feedback - Fully customizable memory module and per-step input structure (no full-history concatenation required)
• Scalable for very long-horizon optimization (context length remains nearly constant over turns)
• Parallelized gym-style environments and group environments (identical initial states for group-based RL)
• Support for various models: Qwen3, Qwen3-VL, Qwen2.5, Qwen2.5-VL, LLaMA3.2, and more - LoRA fine-tuning support (7B models trainable on 2 H100 GPUs)
• Vision-language agent support for multi-modal reasoning - Rich suite of environments: ALFWorld, WebShop, Search (Tool Calling), Sokoban, Gym Cards, AppWorld (experimental)
• Diverse RL algorithms: GiGPO, GRPO, PPO, DAPO, GSPO, RLOO, REINFORCE++, with dynamic sampling and clip-higher variants

Use cases

• Training LLM/VLM agents for long-horizon interactive tasks (e.g., embodied AI, web navigation, tool calling)
• Multi-turn reinforcement learning with customizable memory and input structures - Visual and text-based reasoning agent development - Research on group-based RL algorithms (e.g., GiGPO, GRPO, DAPO)

Getting started

See the repository README for installation instructions and run examples.

License and pricing

The license is Apache-2.0. The project is open source.

Based on the project's public README and repository metadata.