Openrlhf Training

This is a Ray-based RLHF framework built for training large models (7B-70B+) with vLLM inference acceleration. It supports PPO, GRPO, RLOO, and DPO algorithms in one package, claiming 2× speedup over DeepSpeedChat through distributed architecture and GPU resource sharing via ZeRO-3. The standout feature is the Hybrid Engine that lets vLLM and DeepSpeed share GPUs through sleep modes, which matters when you're running actor, critic, reward, and reference models simultaneously. Choose GRPO if you want to skip the critic model and save memory, or stick with PPO for maximum control. The setup is Docker-heavy and Ray-centric, so expect some infrastructure overhead compared to simpler frameworks like TRL, but that's the trade-off for scaling to multi-node clusters.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill openrlhf-training --agent claude-code

Installs into .claude/skills of the current project.

Files

OpenRLHF - High-Performance RLHF Training

Quick start

OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.

Installation:

# Launch Docker container
docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN \
  -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:25.02-py3 bash

# Uninstall conflicts
sudo pip uninstall xgboost transformer_engine flash_attn pynvml -y

# Install OpenRLHF with vLLM
pip install openrlhf[vllm]

PPO Training (Hybrid Engine):

ray start --head --node-ip-address 0.0.0.0 --num-gpus 8

SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthly Promote your product hereReach 125k+ visitors monthly

Agent CaffeineSponsoredClose your lid, come back to finished work

SnugSponsoredThe clipboard manager for the AI era Promote your product hereReach 125k+ visitors monthly

WaxSponsoredBlock anything with supervision, without erasing your iPhone Promote your product hereReach 125k+ visitors monthly

knobsSponsoredSee your site on any phone, any day, in any language

Details

Repository: github.com/orchestra-research/ai-research-skills
Categories: Code Review & Quality, AI & Agent Building, Data Science & ML, Debugging
First seen: Jun 3, 2026