Mcp Eval Runner
A testing harness that lets you write eval fixtures as YAML files and run them directly from your MCP client. Each fixture defines steps with tool calls, inputs, and assertions like output_contains, schema_match, or latency_under. It has two modes: live mode spawns a real MCP server via stdio and tests against actual tool responses, while simulation mode runs assertions against static expected_output strings. You get tools like run_suite for executing all tests, regression_report to compare runs, and create_test_case for scaffolding new fixtures. Step outputs can pipe into downstream inputs using template syntax. Useful when you're building MCP servers and need regression tests in version control without leaving your editor.
- tools
- 10
- auth
- Secret needed
- transport
- STDIO
- env var
- 1
- last updated
- Nov 4, 2026
Tools
10 toolsPublic tool metadata for what this MCP can expose to an agent.
run_suiteexecute all fixtures in the fixtures directory; returns a pass/fail summary
execute all fixtures in the fixtures directory; returns a pass/fail summary
No parameter schema in public metadata yet.
run_caserun a single named fixture by name
run a single named fixture by name
No parameter schema in public metadata yet.
list_casesenumerate available fixtures with step counts and descriptions
enumerate available fixtures with step counts and descriptions
No parameter schema in public metadata yet.
create_test_casecreate a new YAML fixture file (simulation mode; no server block)
create a new YAML fixture file (simulation mode; no server block)
No parameter schema in public metadata yet.
scaffold_fixturegenerate a boilerplate fixture with placeholder steps and pre-filled assertion comments
generate a boilerplate fixture with placeholder steps and pre-filled assertion comments
No parameter schema in public metadata yet.
regression_reportcompare the current fixture state to the last run; surfaces regressions and fixes
compare the current fixture state to the last run; surfaces regressions and fixes
No parameter schema in public metadata yet.
compare_resultsdiff two specific runs by run ID
diff two specific runs by run ID
No parameter schema in public metadata yet.
generate_html_reportgenerate a single-file HTML report for a completed run
generate a single-file HTML report for a completed run
No parameter schema in public metadata yet.
evaluate_deployment_gateCI gate; fails if recent pass rate drops below a configurable threshold
CI gate; fails if recent pass rate drops below a configurable threshold
No parameter schema in public metadata yet.
discover_fixturesdiscover fixture files across one or more directories (respects FIXTURE_LIBRARY_DIRS)
discover fixture files across one or more directories (respects FIXTURE_LIBRARY_DIRS)
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageConfiguration
YOUR_API_KEY*secretYour API key for the service
Readme
MCP Eval Runner
npm mcp-eval-runner package
A standardized testing harness for MCP servers and agent workflows. Define test cases as YAML fixtures (steps → expected tool calls → expected outputs), run regression suites directly from your MCP client, and get pass/fail results with diffs — without leaving Claude Code or Cursor.
Tool reference | Configuration | Fixture format | Contributing | Troubleshooting | Design principles
Key features
Details
- Repository
- github.com/dbsectrainer/mcp-eval-runner
- Category
- AI & LLM Tools
- Package
mcp-eval-runner- Transport
- STDIO
- Auth
- Required
- Registry
- MCP registry, active
More from dbsectrainer
Related AI & LLM Tools MCP servers
View all
Allagiovickhunter/allagio-pluginAgent-to-agent partnerships on ORP: find counterparties, negotiate deals, settle.0stars
Slipstack PDFdev.slipstack/pdfGenerate invoice and receipt PDFs from JSON. Agent-callable document API with a free demo quota.0stars
Sunrays — Local Service Bookingdev.sunrays/bookingBook local tradespeople — plumber, electrician, HVAC, and 7 more — via your AI agent. All US.0stars
Mcp Servertrue402/mcp-serverPay-per-call AI + web + on-chain tools over x402 (USDC on Base): LLM, SEO, extract, token safety.0stars
Apollo Salesforce Mapperdev.workers.selbyventurecap.cf-worker/apollo-salesforce-mapperApollo->Salesforce Lead mapper. No LLM. Strict-fail required fields. PASS/REPAIR/FAIL verdict.0stars
Auto Skill Loaderdivitkashyap/auto-skill-loaderAuto-loads skills into agent sessions. MiniMax vision and search proxy.0stars