Eval Harness

This is a framework for running evals on Claude Code sessions before, during, and after development. You define pass/fail criteria upfront (capability evals for new features, regression evals to catch breakage), then measure reliability with pass@k metrics like "success within 3 attempts." It supports code-based graders for deterministic checks, model graders for open-ended evaluation, and human review flags for security-critical changes. The workflow is straightforward: define evals in `.claude/evals/`, run checks during implementation, generate reports. If you're shipping anything non-trivial with AI assistance and want reproducible quality gates instead of vibes-based shipping, this gives you the scaffolding. The pass@3 threshold of 90% is reasonable for most features.

skills.sh installs: 5.7K
GitHub stars: 229.1K
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add affaan-m/everything-claude-code --skill eval-harness --agent claude-code

Installs into .claude/skills of the current project.

Files

Eval Harness Skill

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.

When to Activate

Setting up eval-driven development (EDD) for AI-assisted workflows
Defining pass/fail criteria for Claude Code task completion
Measuring agent reliability with pass@k metrics
Creating regression test suites for prompt or agent changes
Benchmarking agent performance across model versions

Philosophy

Eval-Driven Development treats evals as the "unit tests of AI development":

Define expected behavior BEFORE implementation
Run evals continuously during development
Track regressions with each change
Use pass@k metrics for reliability measurement

Eval Types

Capability Evals

Test if Claude can do something it couldn't before:

[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
  - [ ] Criterion 1
  - [ ] Criterion 2
  - [ ] Criterion 3
Expected Output: Description of expected result

SKILL.md on GitHub