Generate Synthetic Data

This is a structured approach to creating test data for LLM pipelines when you don't have enough real examples. You define dimensions where your system might fail (like query ambiguity or user persona), generate tuples of those dimensions, then convert them to natural language queries. The two-step separation between tuples and queries is the key move here. It avoids the repetitive, happy-path garbage you get from just prompting "give me test queries." Target is around 100 diverse traces to find failure patterns. Skip this if you already have 100+ real queries, use stratified sampling instead. Also skip it for specialized domains where LLMs can't generate realistic content, like legal filings or low-resource languages.

skills.sh installs: 723
GitHub stars: 1.7K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add hamelsmu/evals-skills --skill generate-synthetic-data --agent claude-code

Installs into .claude/skills of the current project.

Files

Generate Synthetic Data

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.

Prerequisites

Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.

Core Process

Step 1: Define Dimensions

Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.

Dimension 1: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 2: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 3: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Example for a real estate assistant:

Feature: what task the user wants
  Values: [property search, scheduling, email drafting]

Client Persona: who the user serves
  Values: [first-time buyer, investor, luxury buyer]

Scenario Type: query clarity
  Values: [well-specified, ambiguous, out-of-scope]

SKILL.md on GitHub