Evaluate Rag

This walks you through evaluating RAG systems by separating retrieval from generation, which is the right approach but easy to skip when you're just trying to ship. It pushes you to do error analysis on traces first, then build eval datasets with synthetic QA pairs or manual curation. The chunking optimization grid search is genuinely useful since chunk size and overlap can swing your Recall@k by 10+ points on the same retriever. The faithfulness versus relevance breakdown for generation eval is clear, and the anti-pattern list (don't use ROUGE, don't skip traces) will save you from common mistakes. Good for when your RAG pipeline works sometimes but you need systematic measurement to find the actual bottleneck.

skills.sh installs: 850
GitHub stars: 1.7K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add hamelsmu/evals-skills --skill evaluate-rag --agent claude-code

Installs into .claude/skills of the current project.

Files

Evaluate RAG

Overview

Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both.
Build a retrieval evaluation dataset: queries paired with relevant document chunks.
Measure retrieval quality with Recall@k (most important for first-pass retrieval).
Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?).
If retrieval is the bottleneck, optimize chunking via grid search before tuning generation.

Prerequisites

Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.

Core Instructions

Evaluate Retrieval and Generation Separately

Measure each component independently. Use the appropriate metric for each retrieval stage:

First-pass retrieval: Optimize for Recall@k. Include all relevant documents, even at the cost of noise.
Reranking: Optimize for Precision@k, MRR, or NDCG@k. Rank the most relevant documents first.