Eval Audit

This audits your LLM evaluation pipeline and tells you what's broken. It walks through six diagnostic areas: whether you did error analysis on actual traces, if your judges are binary or using noisy Likert scales, whether judges are validated against human labels with TPR/TNR, if you're using similarity metrics like ROUGE as primary evals, how your human review process works, and if you have enough labeled data. The output is a prioritized findings report with concrete next steps linked to other skills. Most useful when you inherit an eval system and don't trust it, or when you have evals running but suspect they're missing real failures. The checks are opinionated but grounded in what actually breaks in production.

skills.sh installs: 1.5K
GitHub stars: 1.7K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add hamelsmu/evals-skills --skill eval-audit --agent claude-code

Installs into .claude/skills of the current project.

Files

Eval Audit

Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.

Overview

Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
Run diagnostic checks across six areas
Produce a findings report ordered by impact, with each finding linking to a fix

Prerequisites

Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."

Connecting to Eval Infrastructure

Check whether the user has an observability MCP server connected (Phoenix, Braintrust, LangSmith, Truesight or similar). If available, use it to pull traces, evaluator definitions, and experiment results. If not, ask for local files: CSVs, JSON trace exports, notebooks, or evaluation scripts.