Validate Evaluator

This walks you through calibrating an LLM judge against human labels using proper train/dev/test splits and TPR/TNR metrics. You'd use it after writing a judge prompt when you need to verify it actually agrees with human judgment before trusting it in production. The workflow is methodical: split your labeled data, iterate on the dev set until you hit 90% TPR and TNR, then measure once on the held-out test set. It includes the Rogan-Gladen bias correction formula for estimating true success rates from biased judge scores, plus bootstrap confidence intervals. The anti-pattern section is worth reading since most people skip validation entirely and just assume judges work.

skills.sh installs: 739
GitHub stars: 1.7K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add hamelsmu/evals-skills --skill validate-evaluator --agent claude-code

Installs into .claude/skills of the current project.

Files

Validate Evaluator

Calibrate an LLM judge against human judgment.

Overview

Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
Run judge on dev set and measure TPR/TNR
Iterate on the judge until TPR and TNR > 90% on dev set
Run once on held-out test set for final TPR/TNR
Apply bias correction formula to production data

Prerequisites

A built LLM judge prompt (from write-judge-prompt)
Human-labeled data: ~100 traces with binary Pass/Fail labels per failure mode
- Aim for ~50 Pass and ~50 Fail (balanced, even if real distribution is skewed)
- Labels must come from a domain expert, not outsourced annotators
Candidate few-shot examples from your labeled data