Iris
If you're shipping AI agents to production and need to know whether they're leaking PII, hallucinating, or burning through your budget, this is the eval layer to add. It logs hierarchical trace trees with per-call latency and token costs to SQLite, runs 13 built-in safety and quality rules (SSN detection, prompt injection patterns, hedge phrase markers), and exposes nine MCP tools including LLM-as-judge scoring and semantic citation verification. Any MCP-compatible agent discovers it automatically once you add the npx command to your config. Flip on the dashboard flag to get a real-time web UI at localhost:6920 showing cost breakdowns and rule pass rates. Bring your own Anthropic or OpenAI key for the judge-based evals.
- tools
- 12
- auth
- Secret needed
- transport
- STDIO
- env vars
- 24
- last updated
- Oct 25, 2026
Tools
12 toolsPublic tool metadata for what this MCP can expose to an agent.
log_traceLog an agent execution with spans, tool calls, token usage, and cost; pass `evaluate: true` to score it in the same call
Log an agent execution with spans, tool calls, token usage, and cost; pass `evaluate: true` to score it in the same call
No parameter schema in public metadata yet.
evaluate_outputScore output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
No parameter schema in public metadata yet.
get_tracesQuery stored traces with filtering, pagination, and time-range support, and find the run where the agent said something with `q`: full-text search over input, output, tool-call values and metadata, ranked, with the matched words marked
Query stored traces with filtering, pagination, and time-range support, and find the run where the agent said something with `q`: full-text search over input, output, tool-call values and metadata, ranked, with the matched words marked
No parameter schema in public metadata yet.
list_rulesEnumerate deployed custom eval rules (read-only)
Enumerate deployed custom eval rules (read-only)
No parameter schema in public metadata yet.
deploy_ruleRegister a new custom eval rule so it fires on every `evaluate_output` of that category
Register a new custom eval rule so it fires on every `evaluate_output` of that category
No parameter schema in public metadata yet.
delete_ruleRemove a deployed custom rule (destructive, idempotent)
Remove a deployed custom rule (destructive, idempotent)
No parameter schema in public metadata yet.
delete_traceRemove a single stored trace by ID (destructive, tenant-scoped)
Remove a single stored trace by ID (destructive, tenant-scoped)
No parameter schema in public metadata yet.
evaluate_with_llm_judgeSemantic eval via LLM (Anthropic or OpenAI). Seven templates: accuracy, helpfulness, safety, correctness, faithfulness, task_completed, relevance. Cost-capped, per-eval pricing disclosed.
Semantic eval via LLM (Anthropic or OpenAI). Seven templates: accuracy, helpfulness, safety, correctness, faithfulness, task_completed, relevance. Cost-capped, per-eval pricing disclosed.
No parameter schema in public metadata yet.
verify_citationsExtract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim.
Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim.
No parameter schema in public metadata yet.
compare_runsCompares two runs of stored evaluations: a paired exact test when the runs share case keys, an interval on the difference otherwise, an honest "cannot tell" with the number of cases it would take, or "equivalent within a margin".
Compares two runs of stored evaluations: a paired exact test when the runs share case keys, an interval on the difference otherwise, an honest "cannot tell" with the number of cases it would take, or "equivalent within a margin".
No parameter schema in public metadata yet.
compare_tracesPer-case pass rates with intervals, flaky cases first, and an overall rate that respects repeats
Per-case pass rates with intervals, flaky cases first, and an overall rate that respects repeats
No parameter schema in public metadata yet.
evaluate_runsRe-score every trace in a run under today's rules into a new run, so a rules change is never read as an agent change
Re-score every trace in a run under today's rules into a new run, so a rules change is never read as an agent change
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageConfiguration
IRIS_API_KEYsecretAPI key for HTTP authentication. Required to bind the HTTP transport or the dashboard beyond loopback (0.0.0.0, a LAN address, a container): without it the server refuses to start
IRIS_API_KEY_FILEPath to a file whose trimmed contents are the API key (the secret-file pattern Docker and Kubernetes mount). Set this or IRIS_API_KEY, not both; further keys and rotation are security.apiKeys in config.json
IRIS_ALLOW_UNAUTHENTICATEDSet to 1 to run a non-loopback bind with NO API key on purpose (lifts the startup refusal; the network is then your boundary)
IRIS_DB_PATHSQLite database path
IRIS_SQLITE_DRIVERWhich SQLite driver holds the database: native (better-sqlite3, the default) or node (Node's built-in node:sqlite, Node 22.13+). Unset: native, falling back to node when the native module cannot load
IRIS_SEARCH_BUDGET_MSHow long one trace search may read before it answers with the matches found so far, in milliseconds (50-60000, default 1000)
IRIS_SEARCH_INDEXon (default) or off. off keeps no full-text index of the traces: writes are cheaper, and a search reads the traces within its time budget
IRIS_LOG_LEVELLog level: debug, info, warn, error
IRIS_HOMEDirectory for the database, custom rules and preferences (default ~/.iris)
IRIS_ANTHROPIC_API_KEYsecretEnables evaluate_with_llm_judge and verify_citations with an Anthropic model. Optional: the deterministic rules never need a key
IRIS_OPENAI_API_KEYsecretEnables evaluate_with_llm_judge and verify_citations with an OpenAI model. Optional: the deterministic rules never need a key
IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVALHard cap on the worst-case cost of one LLM judge call (default 0.25); max_cost_usd can lower it for a call, never raise it
IRIS_RELEVANCE_JUDGE_MODELA priced judge model; answers_the_ask then asks it on every evaluation with input, sending that input and output to the model's provider, and gates on its verdict (off by default)
IRIS_LLM_JUDGE_DAILY_BUDGET_USDWhat every judge call together may spend per UTC day, per tenant, in USD (default 1): the relevance judge, evaluate_with_llm_judge and verify_citations. Past it no judge call is made
IRIS_RELEVANCE_JUDGE_DAILY_BUDGET_USDThe name of IRIS_LLM_JUDGE_DAILY_BUDGET_USD before 0.21.0. Read when the new name is unset, and it then limits every judge call
IRIS_RELEVANCE_JUDGE_MAX_CALLS_PER_REQUESTRelevance judge calls one request may make (default 20)
IRIS_RELEVANCE_JUDGE_REDACTon (default) replaces the PII and credentials no_pii flags before the relevance judge sends input and output to its provider; off sends them as they are
IRIS_CITATION_ALLOW_FETCHLet verify_citations fetch URLs (off by default); a tool argument can turn it off for a call, never on
IRIS_DASHBOARDStart the dashboard alongside the server (same as --dashboard)
IRIS_DASHBOARD_PORTDashboard port (default 6920); IRIS_PORT is the MCP HTTP transport port (default 3000)
IRIS_PORTMCP HTTP transport port (default 3000); only used with --transport http
IRIS_OTEL_ENDPOINTOTLP/HTTP collector URL; when set, every stored trace is also exported as OpenTelemetry spans (best-effort, asynchronous)
IRIS_CITATION_DOMAINSComma-separated domain allowlist for verify_citations fetches (only consulted when IRIS_CITATION_ALLOW_FETCH is on); a tool argument can narrow it, never add to it
IRIS_CITATION_MAX_COST_USD_TOTALThe most one verify_citations call may spend on judge calls, in USD (default 1); max_cost_usd_total can lower it, never raise it
Readme
Details
- Repository
- github.com/iris-eval/mcp-server
- Website
- iris-eval.com
- Category
- AI & LLM Tools
- Package
@iris-eval/mcp-server- Transport
- STDIO
- Auth
- Required
- Registry
- MCP registry, active
Related AI & LLM Tools MCP servers
View all
Lightning Enable MCP Serverrefined-element/lightning-enable-mcpAI agent Lightning payments: L402, invoices, wallets, API discovery, budgets, self-serve signup.9stars
Mementoscrypster/mementoPersistent memory for AI tools. Local-first knowledge graph with hybrid search.9stars
Mcp Context Serveralex-feel/mcp-context-serverAn MCP server that provides persistent multimodal context storage for LLM agents.8stars
Citation Intelligenceautomatelab-tech/citation-intelligenceCheck what Perplexity, Claude, ChatGPT, Gemini, and Google AI Overviews cite for any query.8stars
Media Gen Mcpstrato-space/media-gen-mcpMCP server for OpenAI Images/Videos and Google GenAI (Veo) media generation.8stars