Okareo
If you're building voice or text agents and need to run them through their paces, this connects Claude to Okareo's full evaluation stack. You can register models, create test scenarios, run evaluations with quality checks, and simulate multi-turn conversations with synthetic users right from your editor. The remote mode handles auth via OAuth and lets you switch between Okareo organizations without bouncing out to the web UI. It also exposes voice monitoring tools to ingest and analyze completed conversations from providers like Twilio or Deepgram. Useful when you want to find edge cases and regression test agents without leaving your workflow.
- tools
- 59
- transport
- HTTP
- last updated
- Nov 12, 2026
Tools
59 toolsPublic tool metadata for what this MCP can expose to an agent.
save_scenarioSave a named scenario from rows of input/result data (idempotent)
Save a named scenario from rows of input/result data (idempotent)
No parameter schema in public metadata yet.
list_scenariosList all scenarios in the project with names, IDs, and row counts
List all scenarios in the project with names, IDs, and row counts
No parameter schema in public metadata yet.
get_scenarioRetrieve a scenario's metadata and all data rows by name or ID
Retrieve a scenario's metadata and all data rows by name or ID
No parameter schema in public metadata yet.
create_scenario_versionCreate a new version of an existing scenario with updated data
Create a new version of an existing scenario with updated data
No parameter schema in public metadata yet.
preview_delete_scenarioPreview what will be deleted before removing a scenario
Preview what will be deleted before removing a scenario
No parameter schema in public metadata yet.
delete_scenarioPermanently delete a scenario and all related test data
Permanently delete a scenario and all related test data
No parameter schema in public metadata yet.
list_available_llmsBrowse available LLMs from the Okareo registry
Browse available LLMs from the Okareo registry
No parameter schema in public metadata yet.
register_generation_modelRegister a generation model for testing by selecting an LLM from the registry
Register a generation model for testing by selecting an LLM from the registry
No parameter schema in public metadata yet.
list_generation_modelsList all registered generation models in the project
List all registered generation models in the project
No parameter schema in public metadata yet.
get_generation_modelRead detailed information about a registered generation model
Read detailed information about a registered generation model
No parameter schema in public metadata yet.
update_generation_modelChange the LLM a registered generation model points to
Change the LLM a registered generation model points to
No parameter schema in public metadata yet.
delete_generation_modelRemove a registered generation model and all its related test data
Remove a registered generation model and all its related test data
No parameter schema in public metadata yet.
list_checksList available quality checks (built-in and custom) for evaluating model outputs
List available quality checks (built-in and custom) for evaluating model outputs
No parameter schema in public metadata yet.
run_testRun a quality test that evaluates a model against a scenario using specified checks
Run a quality test that evaluates a model against a scenario using specified checks
No parameter schema in public metadata yet.
list_test_runsFind past test runs, most recent first. detail_level="summary" (default) or "detailed" for aggregate metrics
Find past test runs, most recent first. detail_level="summary" (default) or "detailed" for aggregate metrics
No parameter schema in public metadata yet.
get_test_run_resultsLoad one page of a run's results by ID or name. detail_level="summary" (default) / "detailed" (per-check outcomes + explanations) / "full" (transcripts). Defaults to 20 conversations. Carries a rerun block — the call and config to re-run this simulation
Load one page of a run's results by ID or name. detail_level="summary" (default) / "detailed" (per-check outcomes + explanations) / "full" (transcripts). Defaults to 20 conversations. Carries a rerun block — the call and config to re-run this simulation
No parameter schema in public metadata yet.
get_conversation_transcriptRetrieve the full conversation transcript for a single data point
Retrieve the full conversation transcript for a single data point
No parameter schema in public metadata yet.
reevaluate_test_runRe-score a completed test run against a (possibly different) set of checks
Re-score a completed test run against a (possibly different) set of checks
No parameter schema in public metadata yet.
create_or_update_checkCreate or update a quality check by name — model-based, code-based, or audio (upsert)
Create or update a quality check by name — model-based, code-based, or audio (upsert)
No parameter schema in public metadata yet.
generate_checkGenerate a check from a natural-language description, then save it
Generate a check from a natural-language description, then save it
No parameter schema in public metadata yet.
get_checkRetrieve a check's full configuration, including its prompt template or code
Retrieve a check's full configuration, including its prompt template or code
No parameter schema in public metadata yet.
delete_checkPermanently delete a check by name
Permanently delete a check by name
No parameter schema in public metadata yet.
calibrate_checkDry-run a draft check against a finished test run — per-row verdicts and the arguments the check received, saving nothing
Dry-run a draft check against a finished test run — per-row verdicts and the arguments the check received, saving nothing
No parameter schema in public metadata yet.
create_or_update_targetCreate or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio)
Create or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio)
No parameter schema in public metadata yet.
get_targetRetrieve a Target's configuration by name (all types)
Retrieve a Target's configuration by name (all types)
No parameter schema in public metadata yet.
list_targetsFind simulation targets (voice and custom_endpoint). Bounded to 20 by default; name_contains filters
Find simulation targets (voice and custom_endpoint). Bounded to 20 by default; name_contains filters
No parameter schema in public metadata yet.
delete_targetRemove a simulation target and all its related test data
Remove a simulation target and all its related test data
No parameter schema in public metadata yet.
create_or_update_driverDefine a simulated user persona that will interact with your target
Define a simulated user persona that will interact with your target
No parameter schema in public metadata yet.
get_driverRetrieve a Driver's full configuration including the persona prompt
Retrieve a Driver's full configuration including the persona prompt
No parameter schema in public metadata yet.
list_driversFind Driver personas. Bounded to 20 by default; name_contains filters
Find Driver personas. Bounded to 20 by default; name_contains filters
No parameter schema in public metadata yet.
list_driver_voicesDiscover the voices, voice profiles, and languages available for voice drivers
Discover the voices, voice profiles, and languages available for voice drivers
No parameter schema in public metadata yet.
run_simulationRun a multi-turn conversation evaluation. With based_on_run_id it inherits the original run's full configuration; any argument you pass overrides that one field
Run a multi-turn conversation evaluation. With based_on_run_id it inherits the original run's full configuration; any argument you pass overrides that one field
No parameter schema in public metadata yet.
list_simulationsList past simulation runs with optional filters (target, scenario, limit)
List past simulation runs with optional filters (target, scenario, limit)
No parameter schema in public metadata yet.
ingest_conversationsSubmit completed voice conversations to Okareo for monitoring
Submit completed voice conversations to Okareo for monitoring
No parameter schema in public metadata yet.
connect_voice_integrationConnect a voice provider so its traffic flows into Okareo monitoring
Connect a voice provider so its traffic flows into Okareo monitoring
No parameter schema in public metadata yet.
list_voice_integrationsList the voice provider integrations in your project
List the voice provider integrations in your project
No parameter schema in public metadata yet.
get_voice_integrationRetrieve a voice provider integration by ID, including its status
Retrieve a voice provider integration by ID, including its status
No parameter schema in public metadata yet.
update_voice_integrationUpdate a voice provider integration's metadata
Update a voice provider integration's metadata
No parameter schema in public metadata yet.
rotate_voice_integration_secretRotate a voice provider integration's secrets
Rotate a voice provider integration's secrets
No parameter schema in public metadata yet.
delete_voice_integrationDelete a voice provider integration by ID
Delete a voice provider integration by ID
No parameter schema in public metadata yet.
get_voice_webhook_urlGet the inbound webhook endpoint for a voice provider
Get the inbound webhook endpoint for a voice provider
No parameter schema in public metadata yet.
list_projectsList every project you can work in — id, name, tags, archive state — with the active one marked
List every project you can work in — id, name, tags, archive state — with the active one marked
No parameter schema in public metadata yet.
select_projectChoose the project to work in for this conversation
Choose the project to work in for this conversation
No parameter schema in public metadata yet.
create_projectCreate a new project
Create a new project
No parameter schema in public metadata yet.
update_projectRename a project, or replace its tags
Rename a project, or replace its tags
No parameter schema in public metadata yet.
archive_projectHide a project from the project picker. Reversible; nothing is deleted
Hide a project from the project picker. Reversible; nothing is deleted
No parameter schema in public metadata yet.
unarchive_projectRestore an archived project to the picker
Restore an archived project to the picker
No parameter schema in public metadata yet.
clone_projectCopy an existing project's scenarios into a new project (dry-run first)
Copy an existing project's scenarios into a new project (dry-run first)
No parameter schema in public metadata yet.
query_analyticsQuery Okareo's product analytics to understand evaluation trends
Query Okareo's product analytics to understand evaluation trends
No parameter schema in public metadata yet.
list_dashboardsList the analytics dashboards in your project
List the analytics dashboards in your project
No parameter schema in public metadata yet.
get_dashboardRetrieve a dashboard's full configuration by name
Retrieve a dashboard's full configuration by name
No parameter schema in public metadata yet.
save_dashboardCreate or update an analytics dashboard by name (upsert)
Create or update an analytics dashboard by name (upsert)
No parameter schema in public metadata yet.
reorder_dashboardsSet the display order of dashboards
Set the display order of dashboards
No parameter schema in public metadata yet.
delete_dashboardDelete a dashboard by name
Delete a dashboard by name
No parameter schema in public metadata yet.
list_tenantsList every Okareo organization you have access to in this MCP session
List every Okareo organization you have access to in this MCP session
No parameter schema in public metadata yet.
switch_tenantDoesn't change the active organization; returns how to reconnect and pick another one at sign-in
Doesn't change the active organization; returns how to reconnect and pick another one at sign-in
No parameter schema in public metadata yet.
get_docsQuery the Okareo documentation system for conceptual or user-legible explanations
Query the Okareo documentation system for conceptual or user-legible explanations
No parameter schema in public metadata yet.
get_templatesRetrieve prompt templates for common Okareo patterns (works offline)
Retrieve prompt templates for common Okareo patterns (works offline)
No parameter schema in public metadata yet.
get_reps_baselineServe REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release
Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageReadme
Okareo MCP Server
The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.
For detailed documentation, see the Okareo MCP docs.
Prerequisites
- An Okareo account at app.okareo.com
- A copilot that supports MCP servers (Claude Code, Cursor, VS Code, or GitHub Copilot), or Slackbot in Slack
- Python 3.10–3.12 (only for the local install modes; not needed for remote)
Remote MCP (hosted) — recommended
The fastest way to get started. No local install, no Python, no API key in .mcp.json. Browser sign-in handles auth on first connect.
Recommended (OAuth — Claude Code, Claude Desktop, Cursor, VS Code 1.101+)
Add to your copilot's MCP config (typically .mcp.json):
{
"mcpServers": {
"okareo": {
"type": "http",
"url": "https://tools.okareo.com/mcp"
}
}
}
Details
- Repository
- github.com/okareo-ai/okareo-mcp
- Transport
- HTTP
- Registry
- MCP registry, active