Error Analysis

Error Analysis

Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.

Overview

1. Collect ~100 representative traces
2. Read each trace, judge pass/fail, and note what went wrong
3. Group similar failures into categories
4. Label every trace against those categories
5. Compute failure rates to prioritize what to fix

Core Process

Step 1: Collect Traces

Capture the full trace: input, all intermediate LLM calls, tool uses, retrieved documents, reasoning steps, and final output.

Target: ~100 traces. This is roughly where new traces stop revealing new kinds of failures. The number depends on system complexity.

From real user data (preferred):

• Small volume: random sample
• Large volume: sample across key dimensions (query type, user segment, feature area)
• Use embedding clustering (K-means) to ensure diversity

From synthetic data (when real data is sparse):

• Use the generate-synthetic-data skill
• Run synthetic queries through the full pipeline and capture complete traces

Step 2: Read Traces and Take Notes