Constitutional Ai

This implements Anthropic's Constitutional AI training method, the two-phase approach that powers Claude's safety system. You generate responses, have the model critique itself against a constitution (set of principles), revise based on critiques, then fine-tune. The RL phase uses RLAIF where AI evaluators replace human labelers for preference data. The real win is scalable safety alignment without needing humans to label harmful outputs, though you'll need to iterate on your constitution principles to avoid overly evasive refusals. Requires meaningful compute for the RL phase (2x A100s for 7B models) since you're training both policy and reward models. Good fit if you want explainable safety decisions and have clear principles to encode.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill constitutional-ai --agent claude-code

Installs into .claude/skills of the current project.

Files

Constitutional AI - Harmlessness from AI Feedback

Quick start

Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.

Key concept: Models learn to critique and revise their own responses using a "constitution" (set of principles).

Two phases:

Supervised Learning (SL): Self-critique + revision
Reinforcement Learning (RL): RLAIF (RL from AI Feedback)

Constitution example:

Principles:
1. Choose the response that is most helpful, honest, and harmless
2. Avoid responses that are toxic, racist, or sexist
3. Prefer responses that explain objections rather than refuse
4. Choose responses that are thoughtful and nuanced

Common workflows

Workflow 1: Supervised learning phase (self-critique + revision)

Step 1: Generate initial responses:

from transformers import pipeline

generator = pipeline("text-generation", model="base-model")

prompts = [
    "How do I hack a website?",
    "Write something offensive about a group.",
    "Help me cheat on my exam."
]

initial_responses = generator(prompts, max_length=200)

SKILL.md on GitHub