Evaluation Methodology

This is the internal reference for how PluginEval measures skill quality across ten dimensions, from triggering accuracy to ecosystem coherence. It documents the three-layer scoring system (static analysis, LLM judge, Monte Carlo simulation), the weighted composite formula that produces final scores, and the badge thresholds from Bronze to Platinum. You'll want this when debugging why a skill scored poorly on orchestration fitness, when calibrating your own marketplace's quality bar, or when explaining to a partner why their integration earned a C on token efficiency. The blend weights table alone is worth bookmarking. Fair warning: this is dense reference material, not a tutorial.

skills.sh installs: 7.6K
GitHub stars: 42.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add wshobson/agents --skill evaluation-methodology --agent claude-code

Installs into .claude/skills of the current project.

Files

Evaluation methodology

PluginEval scores a skill or a plugin from 0 to 100 by combining up to three layers. The static layer is a lint. It's fast and deterministic, and it's useful for checking structure. The LLM judge and Monte Carlo layers are experimental, not validated against human labels, so treat their numbers as rough signals. For the trace-based eval program, see evals/README.md at the repository root.

The judge rubric anchors are in references/rubrics.md. Fixes for each anti-pattern and tips for each dimension are in references/improving-scores.md.

Evaluation depths

Depth	Layers	Confidence label
`quick`	static	Estimated
`standard` (default for `score`)	static and judge	Assessed
`deep` (used by `certify`)	static, judge, and Monte Carlo with 50 runs	Certified
`thorough`	static, judge, and Monte Carlo with 100 runs	Certified+