OpenRLHF

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

What is OpenRLHF?

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

Overview of OpenRLHF

What it is

OpenRLHF is an open-source framework for reinforcement learning from human feedback (RLHF). It is built on a distributed architecture combining Ray, vLLM, and DeepSpeed. The framework is designed to be easy to use, scalable, and high-performance, supporting agentic RL training with algorithms such as PPO, DAPO, REINFORCE++, and others. It is licensed under Apache-2.0.

Key features

  • Built on Ray + vLLM distributed architecture for efficient scheduling and high-throughput inference.
  • Supports a unified agent-based execution paradigm with token-in-token-out processing for consistency across single-turn and multi-turn tasks.
  • Includes RL algorithms: PPO, REINFORCE++, GRPO, RLOO, and FlashREINFORCE.
  • Provides hybrid engine scheduling to share GPU resources among models and vLLM engines, minimizing idle time.
  • Uses DeepSpeed ZeRO-3 for memory-efficient training and integrates with HuggingFace Transformers.
  • Supports vision-language model (VLM) RLHF and multi-turn VLM RL with image inputs.
  • Offers async RLHF training and agent RLHF via configuration flags.

Use cases

OpenRLHF can be used for standard RLHF alignment of large language models, training reasoning models with prolonged RL (e.g., REINFORCE++-baseline), multi-turn agentic tasks with custom reward functions, and multimodal RLHF involving images. It is also suitable for reproducing research results such as DeepSeek-R1 on small models.

Getting started

See the repository README for installation and typical workflow commands.

License and pricing

OpenRLHF is open-source under the Apache-2.0 license.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/openrlhfopenrlhf.svg)](https://yunodb.com/tools/openrlhfopenrlhf?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/openrlhfopenrlhf?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/openrlhfopenrlhf.svg" alt="Featured on YunoDB" height="20"></a>