Benchmark E2e
This is an overnight CI pipeline for the vercel-plugin project. It spins up nine realistic projects (recipe platform, trivia game, code review bot), exercises skill injection through claude --print, verifies dev servers actually boot, then diffs which skills were expected versus which actually got injected. The output is a machine-readable report with copy-pasteable YAML fixes for missing patterns. Run it in a loop with sleep 3600 between iterations and you get a self-improvement cycle that shows exactly which gaps closed since the last run. Honestly feels like internal tooling that escaped into the wild, but if you're building a Claude plugin with similar injection logic, the contracts (run-manifest.json, events.jsonl) are worth stealing.
- skills.sh installs
- 187
- GitHub stars
- 183
- last updated
- Jun 3, 2026
Install
$ npx -y skills add vercel-labs/vercel-plugin --skill benchmark-e2e --agent claude-codeInstalls into .claude/skills of the current project.
Files
Benchmark E2E
Single-command pipeline that creates projects, exercises skill injection via claude --print, launches dev servers, verifies they work, analyzes conversation logs, and generates actionable improvement reports.
Quick Start
# Full suite (9 projects, ~2-3 hours)
bun run scripts/benchmark-e2e.ts
# Quick mode (first 3 projects, ~30-45 min)
bun run scripts/benchmark-e2e.ts --quick
Options:
| Flag | Description | Default |
|---|---|---|
--quick | Run only first 3 projects | false |
--base <path> | Override base directory | ~/dev/vercel-plugin-testing |
--timeout <ms> | Per-project timeout (forwarded to runner) | 900000 (15 min) |
Pipeline Stages
The orchestrator chains four stages sequentially, aborting on failure:
- runner — Creates test dirs, installs plugin, runs
claude --printwithVERCEL_PLUGIN_LOG_LEVEL=trace - verify — Detects package manager, launches dev server, polls for 200 with non-empty HTML
- analyze — Matches JSONL sessions to projects via
run-manifest.json, extracts metrics - report — Generates
report.mdandreport.jsonwith scorecards and recommendations
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/vercel-labs/vercel-plugin
- Categories
- Testing & QA, Cloud & Infrastructure
- First seen
- Jun 3, 2026
More from vercel-labs/vercel-plugin
All 32- Benchmark Sandbox181
- Next Forge158
- Nextjs4.3K
- React Best Practices721
- Shadcn399
- Deployments Cicd333
- Turbopack310
- Ai Sdk278
- Vercel Cli272
- Routing Middleware264
- Vercel Agent263
- Ai Gateway260
- Vercel Functions259
- Chat Sdk257
- Env Vars250
- Vercel Storage244
- Workflow236
- Auth232
- Runtime Cache228
- Verification226
- Vercel Sandbox219
- Next Cache Components218
- Next Upgrade213
- Bootstrap211
Recommended
More Testing & QA
smtp-penetration-testingsickn33/antigravity-awesome-skillssmtp penetration testing186skills.sh installs
alicloud-network-alb-testcinience/alicloud-skillsSmoke test for Alibaba Cloud ALB skill. Validates SDK auth, script compilation, list instances, and health check flows.184skills.sh installs
ln-637-test-structure-auditorlevnikolaevich/claude-code-skillsAudits test maintainability and structure: layout drift, orphan tests, fragmented duplicates, test-to-source mapping, and domain grouping. Use when auditing tes184skills.sh installs
html-ppt-testing-safety-alertnexu-io/open-design红琥珀警示 deck — 顶/底 45° 红黑 hazard 条纹、红色删除线否定标题、L1/L2/L3 绿/琥珀/红 tier 卡片、圆点状态 alert box、policy-yaml 代码块（红左边框 + bad 关键词高亮）、红绿 checklist、Q1 事故堆叠柱状图。适合安全 / 风险 / 事故复盘 / 182skills.sh installs
n8n-security-testingproffesor-for-testing/agentic-qeCredential exposure detection, OAuth flow validation, API key management testing, and data sanitization verification for n8n workflows. Use when validating n8n 172skills.sh installs