Model Evaluation

Automates the process of evaluating SageMaker models by generating Python code for two evaluation approaches: LLM-as-Judge, where another LLM grades your model's outputs, and Custom Scorer, which uses Lambda functions for programmatic testing like math or code validation. It works with both open source models and Nova, though LLM-as-Judge is OSS only. The workflow is conversational and deliberate, asking one question at a time to determine which evaluation type fits your situation, validating prerequisites like dataset availability and model compatibility, then generating the actual evaluation code. If you're running benchmarks or comparing fine-tuned models in SageMaker, this handles the boilerplate so you can focus on interpreting results instead of wiring up evaluation infrastructure.

skills.sh installs: 140
GitHub stars: 938
last updated: Jun 11, 2026

View on GitHub

Install

$ npx -y skills add awslabs/agent-plugins --skill model-evaluation --agent claude-code

Installs into .claude/skills of the current project.

Files

Model Evaluation

Generate code that evaluates a SageMaker model.

Prerequisites

The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Principles

One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.
Confirm before proceeding. Wait for the user to agree before moving to the next step.
Don't read files until you need them. Only read reference files when you've reached the step that requires them.
Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.
No narration. Share outcomes and ask questions. Keep responses short.
No repetition. If you said something before a tool call, don't repeat it after.

Scope

This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.

SKILL.md on GitHub