agent-regression-evals

alirezarezvani/claude-skills

Prove an agent, prompt, model or tool change did not make the agent worse. Three deterministic stdlib scripts: a trajectory checker that grades tool-call paths (strict / unordered / subset / superset match modes, argument matching, forbidden tools that block in every mode, call ceilings); a flake detector that classifies each case as stable or flaky from repeated trials and reports pass@k and pass^k with Wilson intervals; and a paired regression gate that averages trials per case, tests per-case differences with a t-based CI and an exact one-sided McNemar test, and blocks the merge on a significant drop, a critical-case regression, a dropped case, or a quarantine entry without owner and expiry. Ships cs-eval-gatekeeper agent, /cs-agent-evals and /cs-eval-gate commands, 3 references, and a CI job example.

version

2.14.5

GitHub stars

29.6K

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language