Vally Eval

microsoft/github-copilot-for-azureOfficial

If you're building agent skills and need proper integration testing, this handles the full lifecycle of Vally eval suites. It'll help you author eval.yaml files, validate they follow conventions like required tags for type and tier, run them with custom executors that support early termination and screenshot capture, and even re-grade existing trajectories when you're tuning grader configs. It's designed for the azure-skills plugin ecosystem where test results feed into monitoring dashboards. The custom executor piece is crucial here since it extends Vally with features the base tool doesn't support yet. Worth noting you can't use the completed grader if you specify early termination conditions, which makes sense but could trip you up.

GitHub stars: 217
last updated: Jun 11, 2026

View on GitHub

Install

$ npx -y skills add microsoft/github-copilot-for-azure --skill vally-eval --agent claude-code

Installs into .claude/skills of the current project.

Files

Vally eval suites

Skills in the azure-skills plugin are required to have integration tests that run prompts against an LLM agent to evaluate whether they help the agent accomplish goals in target scenarios. Such integration tests are written as vally eval suites, using vally as the underlying tool for running tests and grading the agent outcome.

Write vally eval suites

Vally eval suites are written as yaml documents. All eval suites share eval spec.

Refer to the official documentation on the schema of the spec and the schema of the eval suites writing-eval-specs.

Vally eval suites for azure-skills plugin have the following file layout. The shared eval spec is located at <repo-root>/.vally.yaml. The eval suites are categorized by plugin and skills. The eval suites for each skill are located at <repo-root>/evals/<plugin-dirname>/<skill-name>/*.yaml, e.g. <repo-root>/evals/azure-skills/azure-ai/eval.yaml.

SKILL.md on GitHub

Details

Repository: github.com/microsoft/github-copilot-for-azure
Categories: Testing & QA, Git & Pull Requests, AI & Agent Building, Cloud & Infrastructure
First seen: Jun 11, 2026