Simpo Training

SimPO is a reference-free preference optimization method that outperforms DPO by 6.4 points on AlpacaEval 2.0 without needing a reference model during training. You use it when you have preference pairs (chosen/rejected responses) and want simpler, more efficient alignment than DPO or PPO. The implementation lives in Hugging Face's alignment-handbook and requires careful tuning of learning rate (3e-7 to 1e-6) and beta (2.0-10.0) parameters. Works on single-node setups with DeepSpeed ZeRO-3, making it accessible if you don't have massive distributed infrastructure. The honest take: if DPO is your baseline for preference alignment, SimPO gives you better results with less complexity, though you still need to babysit hyperparameters to avoid divergence.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill simpo-training --agent claude-code

Installs into .claude/skills of the current project.

Files

SimPO - Simple Preference Optimization

Quick start

SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.

Installation:

# Create environment
conda create -n simpo python=3.10 && conda activate simpo

# Install PyTorch 2.2.2
# Visit: https://pytorch.org/get-started/locally/

# Install alignment-handbook
git clone https://github.com/huggingface/alignment-handbook.git
cd alignment-handbook
python -m pip install .

# Install Flash Attention 2
python -m pip install flash-attn --no-build-isolation

Training (Mistral 7B):

ACCELERATE_LOG_LEVEL=info accelerate launch \
  --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py \
  training_configs/mistral-7b-base-simpo.yaml

Common workflows

Workflow 1: Train from base model (Mistral 7B)

Config (mistral-7b-base-simpo.yaml):

# Model
model_name_or_path: mistralai/Mistral-7B-v0.1
torch_dtype: bfloat16

# Dataset
dataset_mixer:
  HuggingFaceH4/ultrafeedback_binarized: 1.0
dataset_splits:
  - train_prefs
  - test_prefs

SKILL.md on GitHub