Dataset Evaluation

This validates your dataset formatting before you kick off a SageMaker fine-tuning job, saving you from the annoying failures that happen when your data schema doesn't match what SFT, DPO, or RLVR expects. It auto-detects file format, checks schema compliance against your chosen model and technique, and tells you straight up if your data is ready or broken. Worth noting that it handles both training and evaluation datasets differently, which matters because they have distinct format requirements. The workflow is smart enough to pull model and strategy from conversation context if you've already set those up, otherwise it'll prompt you to select them first. One solid feature is that it won't unnecessarily copy or download S3 data, it just validates in place.

skills.sh installs: 156
GitHub stars: 938
last updated: Jun 11, 2026

View on GitHub

Install

$ npx -y skills add awslabs/agent-plugins --skill dataset-evaluation --agent claude-code

Installs into .claude/skills of the current project.

Files

Workflow Instruction

Follow the workflow shown below. Locate the dataset, check the file type, and resolve any issues with missing files or wrong file types. Determine the fine-tuning model and fine-tuning strategy. Run the appropriate validation based on the model family. Summarize the results: is the dataset ready for fine-tuning?

Prerequisites

The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Workflow

Locate Dataset:
- The full path may be a local file path, or an S3 URI
- Resolve the full path to the dataset file, make sure read permissions are available, and help the user if the file is not found

SKILL.md on GitHub