Ai Evals

This helps you design and implement systematic evaluations for AI products, whether you're measuring LLM output quality, building test cases, or creating scoring rubrics. It pushes you through the practitioner workflow: understanding failure modes through manual review, open coding what's broken, clustering patterns, then writing specific binary criteria. The approach is grounded in insights from folks like Hamel Husain and Shreya Shankar who argue evals are becoming a core product skill, not just an ML engineering concern. Useful when you need to move beyond "does this feel good" to actually measuring whether your AI feature works, and it'll flag common mistakes like skipping manual trace analysis or using fuzzy Likert scales instead of clear pass/fail criteria.

skills.sh installs: 2.2K
GitHub stars: 1.5K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add refoundai/lenny-skills --skill ai-evals --agent claude-code

Installs into .claude/skills of the current project.

Files

AI Evaluation Strategy

Move beyond vibe checks to systematic, empirical measurement of AI product quality and reliability.

Help the user with ai evaluation strategy using insights from 11 guests and posts across Lenny's Podcast and Newsletter.

How to Help

Identify Failure Modes - Help the user conduct error analysis on real traces to find where the system specifically breaks.
Select Eval Methods - Recommend the right mix of human, code, and LLM judges based on the specific technical use case.
Build Gold Sets - Assist in curating a reference dataset of high-quality examples to act as the ground truth for your application.
Operationalize - Guide the user in integrating these evaluations into a CI/CD pipeline for continuous quality improvement.