Agent Eval

A systematic way to compare coding agents on your actual codebase instead of vibes. You define tasks in YAML with a prompt and judge criteria (tests, grep patterns, or LLM eval), then run multiple agents in isolated git worktrees and collect pass rate, cost, time, and consistency metrics. The honest take: this is what you'd build yourself after the third time someone asks "should we use Claude Code or Aider?" and you realize anecdotes don't cut it. Run three trials per agent on five real tasks from your workflow, pin the commit for reproducibility, and you'll have data to make the call. No Docker required, just git worktrees and a willingness to treat agent selection like any other engineering decision.

skills.sh installs: 5.5K
GitHub stars: 300.3K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add affaan-m/ecc --skill agent-eval --agent claude-code

Installs into .claude/skills of the current project.

Files

Agent Eval Skill

A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.