Claude MarketplacesSubmit
tjacquesson avatar

Llmtest Mcp

tjacquesson/llmtest-mcp

Connects Claude and other MCP clients to LLMTest's proxy and benchmarking API to help you find cheaper AI models that still pass your quality bar. Exposes tools to check proxy status, list flows with cost and latency stats, run benchmarks across 340+ models, get and accept cost-saving suggestions, and manage autopilot optimization with drift detection and auto-revert. You point your production app at llmtest.io/v1 instead of calling OpenAI or Anthropic directly, then ask your AI assistant to analyze usage and recommend switches. Pre-launch you can seed test samples and benchmark before any real traffic hits. Includes prompt rewriting to find even cheaper alternatives.

tools
15
auth
Secret needed
transport
STDIO
env var
1
last updated
May 22, 2026
View on GitHub

Tools

15 tools

Public tool metadata for what this MCP can expose to an agent.

statusShow proxy status and activity summary

Show proxy status and activity summary

No parameter schema in public metadata yet.

list_flowsList all AI flows with cost and latency stats

List all AI flows with cost and latency stats

No parameter schema in public metadata yet.

get_suggestionsGet pending model-switch recommendations

Get pending model-switch recommendations

No parameter schema in public metadata yet.

update_suggestionAccept or dismiss a suggestion

Accept or dismiss a suggestion

No parameter schema in public metadata yet.

run_benchmarkBenchmark a flow against challenger models

Benchmark a flow against challenger models

No parameter schema in public metadata yet.

optimize_promptRewrite a flow's prompt and find a cheaper model that still works

Rewrite a flow's prompt and find a cheaper model that still works

No parameter schema in public metadata yet.

seed_samplesAdd test prompts for pre-launch benchmarking

Add test prompts for pre-launch benchmarking

No parameter schema in public metadata yet.

list_samplesShow stored test samples per flow

Show stored test samples per flow

No parameter schema in public metadata yet.

list_new_modelsShow new and trending models

Show new and trending models

No parameter schema in public metadata yet.

get_accountCheck credit balance and usage

Check credit balance and usage

No parameter schema in public metadata yet.

get_autopilot_statusCheck whether autopilot is on and whether the account is eligible

Check whether autopilot is on and whether the account is eligible

No parameter schema in public metadata yet.

enable_autopilotTurn on weekly auto-optimization with safety gates + drift-based auto-revert

Turn on weekly auto-optimization with safety gates + drift-based auto-revert

No parameter schema in public metadata yet.

disable_autopilotTurn off autopilot (existing optimizations stay active)

Turn off autopilot (existing optimizations stay active)

No parameter schema in public metadata yet.

list_active_optimizationsList auto-accepted optimizations still inside their 24h revert window

List auto-accepted optimizations still inside their 24h revert window

No parameter schema in public metadata yet.

revert_optimizationRoll an auto-accepted optimization back to the previous prompt

Roll an auto-accepted optimization back to the previous prompt

No parameter schema in public metadata yet.

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Configuration

LLMTEST_API_KEY*secret

Your LLMTest API key from https://llmtest.io/dashboard

Readme

LLMTest MCP Server

npm version MIT License

MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.

Quick Start

1. Get your API key

Sign up at llmtest.io and grab your API key from the dashboard.

2. Add to your tool

Claude Code:

claude mcp add llmtest -- npx llmtest-mcp

Then set your key:

export LLMTEST_API_KEY=llmt_your_key_here

Cursor / Windsurf / Other MCP clients:

Add to your MCP config file:

{
  "mcpServers": {
    "llmtest": {
      "command": "npx",
      "args": ["llmtest-mcp"],
      "env": {
        "LLMTEST_API_KEY": "llmt_your_key_here"
      }
    }
  }
}

3. Talk to your AI

Just ask in natural language:

  • "Check my LLMTest status"
  • "Find cheaper models for my AI calls"
  • "Run a benchmark on my blog-writer flow"
  • "What models are trending?"

View the full README on GitHub

Details

Repository
github.com/tjacquesson/llmtest-mcp
Website
llmtest.io
Package
llmtest-mcp
Transport
STDIO
Auth
Required
Registry
MCP registry, active