Claude MarketplacesSubmit
launchdarkly avatar

Online Evals

launchdarkly/agent-skillsOfficial

This lets you attach LLM judges to LaunchDarkly config variations to automatically score response quality. You can use built-in judges for accuracy, relevance, and toxicity, or create custom ones for domain-specific evaluation like security or brand compliance. The workflow is straightforward: create judge configs with evaluation prompts, attach them to variations via API, and set sampling rates. One gotcha is that the judges array replaces all existing attachments, so you need to include everything each time. Only works with completion mode configs in the UI, though you can evaluate agent mode programmatically. Requires Python SDK 0.20.0+ or Node.js 0.20.0+ for the consolidated metric recording API.

skills.sh installs
1.3K
GitHub stars
19
last updated
Jul 14, 2026
View on GitHub

Install

$ npx -y skills add launchdarkly/agent-skills --skill online-evals --agent claude-code

Installs into .claude/skills of the current project.

Files

Config Online Evaluations

Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.

Prerequisites

  • LaunchDarkly account with AgentControl enabled
  • The remotely hosted LaunchDarkly MCP server configured in your environment
  • Existing config with variations (use configs-create skill)
  • To run judges from your application: launchdarkly-ai-server 0.3.0+ (Python) or @launchdarkly/ai-node 0.3.0+ (Node.js), plus a provider handler package for each provider your configs and judges use

API Access

Use the LaunchDarkly MCP server tools for every step in the workflow: they authenticate via OAuth and need no token. The REST API is only for scripts and CI that can't use MCP (see Scripting without MCP). In that case, ask the user to paste an API access token for this session and substitute it for {api_token} in the examples below. Do not search environment variables, .env files, or agent/MCP config files for credentials.

Core Concepts

What Are Judges?

SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Details

Repository
github.com/launchdarkly/agent-skills
Category
AI & Agent Building
First seen
Jul 14, 2026

More from launchdarkly/agent-skills

All 41

Recommended

More AI & Agent Building
1launchdarkly avatarprojectslaunchdarkly/agent-skillsprojects1.3Kskills.sh installs
2charleswiltgen avataraxiom-securitycharleswiltgen/axiomUse when storing credentials securely, encrypting data, implementing passkeys, securing AI/agentic features against prompt injection, code signing, or managing 1.3Kskills.sh installs
3neolabhq avatardo-and-judgeneolabhq/context-engineering-kitExecute a task with sub-agent implementation and LLM-as-a-judge verification with automatic retry loop1.3Kskills.sh installs
4nvidia avatarnemoclaw-user-guidenvidia/skillsGuides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, o1.3Kskills.sh installs
5brave avataranswersbrave/brave-search-skillsUSE FOR AI-grounded answers via OpenAI-compatible /chat/completions. Two modes: single-search (fast) or deep research (enable_research=true, thorough multi-sear1.3Kskills.sh installs
6cursor avatarcursor-sdkcursor/pluginsGuide users building apps, scripts, CI pipelines, or automations on top of the Cursor TypeScript SDK (`@cursor/sdk`). Use this skill whenever the user mentions 1.3Kskills.sh installs