Claude MarketplacesSubmit
vercel-labs avatar

Benchmark Agents

vercel-labs/vercel-pluginOfficial

This is a meta-skill for stress-testing the Vercel plugin itself, not for building apps. It walks you through spawning multiple Claude Code sessions in WezTerm panes, each running a different project prompt, then checking debug logs and claim directories to verify which skills got injected. The whole point is validating that PreToolUse hooks fire correctly, that PostToolUse catches validation errors, and that the plugin's skill injection logic works across complex scenarios involving Workflow DevKit, AI Gateway, MCP, and multi-agent orchestration. You run the evals manually in your conversation using bash tool calls, no scripts allowed, because only interactive sessions trigger the plugin hooks. Useful if you're iterating on the plugin's skill catalog or debugging why certain patterns aren't getting picked up.

skills.sh installs
197
GitHub stars
183
last updated
Jun 3, 2026
View on GitHub

Install

$ npx -y skills add vercel-labs/vercel-plugin --skill benchmark-agents --agent claude-code

Installs into .claude/skills of the current project.

Files

Benchmark Agents — Advanced AI Systems

Launch real Claude Code sessions with the plugin installed, verify skill injection, monitor PostToolUse validation catches, and produce a coverage report. This skill covers the full eval loop: setup → launch → monitor → verify → fix → release → repeat.

How Evals Work (The Only Correct Method)

Evals are run by you, in this conversation, not by scripts. The process is:

  1. You create directories and install the plugin via Bash tool calls
  2. You spawn WezTerm panes with wezterm cli spawn — each pane runs an independent Claude Code interactive session
  3. You wait, then check debug logs and claim dirs to see what the plugin injected
  4. You inspect the generated source code for correctness
  5. You read conversation logs to find what the user had to correct
  6. You update skills/hooks, run /release, and spawn more evals
SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Details

Repository
github.com/vercel-labs/vercel-plugin
Category
Cloud & Infrastructure
First seen
Jun 3, 2026

More from vercel-labs/vercel-plugin

All 32

Recommended

More Cloud & Infrastructure
1vercel-labs avatarreleasevercel-labs/vercel-pluginrelease194skills.sh installs
2vercel-labs avatarbenchmark-testingvercel-labs/vercel-pluginbenchmark testing193skills.sh installs
3danielmiessler avatarFirstPrinciplesdanielmiessler/personal_ai_infrastructurefirstprinciples190skills.sh installs
4vercel-labs avatarknowledge-updatevercel-labs/vercel-pluginknowledge update190skills.sh installs
5itsmostafa avatarcognitoitsmostafa/aws-agent-skillsAWS Cognito user authentication and authorization service. Use when setting up user pools, configuring identity pools, implementing OAuth flows, managing user a188skills.sh installs
6vercel-labs avatarbenchmark-e2evercel-labs/vercel-pluginbenchmark e2e187skills.sh installs