Golden Dataset

This handles the full lifecycle of building and maintaining evaluation datasets for AI systems. You get patterns for curating content with multi-agent annotation pipelines, versioning datasets as JSON (not SQL dumps), and validating quality with duplicate detection and drift analysis. The standout piece is the 9-phase workflow for adding new documents that includes parallel quality scoring and bias detection. It's opinionated about thresholds: 0.70 quality score minimum, at least 2 domain tags per entry, and specific difficulty distribution requirements. If you're serious about eval datasets rather than just throwing examples in a folder, the structure here will save you from the usual mess of placeholder URLs and missing referential integrity.

skills.sh installs: 193
GitHub stars: 301
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add yonatangross/orchestkit --skill golden-dataset --agent claude-code

Installs into .claude/skills of the current project.

Files

Golden Dataset

Comprehensive patterns for building, managing, and validating golden datasets for AI/ML evaluation. Each category has individual rule files in rules/ loaded on-demand.

Quick Reference

Category	Rules	Impact	When to Use
Curation	2	HIGH	Content collection, annotation pipelines
Management	2	HIGH	Versioning, backup/restore
Validation	1	CRITICAL	Regression testing
Add Workflow	1	HIGH	9-phase curation, quality scoring, bias detection, silver-to-gold

Total: 6 rules across 4 categories. House thresholds and scars: references/ork-delta.md.

Curation

Content collection, multi-agent annotation, and diversity analysis for golden datasets.

Rule	File	Key Pattern
Collection	`rules/curation-collection.md`	Content type classification, quality thresholds, duplicate prevention
Annotation	`rules/curation-annotation.md`	Multi-agent pipeline, consensus aggregation, Langfuse tracing

SKILL.md on GitHub