Aiconfig Online Evals
This lets you attach LLM judges to AI Config variations for automatic quality scoring. The judges are specialized AI Configs that evaluate responses and return scores from 0.0 to 1.0, measuring things like accuracy, relevance, or toxicity. You can use the three built-in judges or create custom ones for domain-specific evaluation like security auditing or contract compliance. The sampling rate controls what percentage of responses get evaluated, which matters if you're running evals at scale. One gotcha: judges only work with completion mode configs through the UI, and you have to manually set the fallthrough variation since the normal targeting toggle doesn't work for AI Configs. Requires Python v0.18.0+ or Node v0.17.0+ for the consolidated judge result API.
- skills.sh installs
- 2.2K
- GitHub stars
- 19
- last updated
- Jul 14, 2026
Install
$ npx -y skills add launchdarkly/agent-skills --skill aiconfig-online-evals --agent claude-codeInstalls into .claude/skills of the current project.
Files
Deprecated — renamed to online-evals
This skill has been renamed. Do not follow any instructions here and do not perform the task from this skill.
Invoke the online-evals skill instead — it contains the current, complete instructions.
If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from aiconfig-online-evals to online-evals.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/launchdarkly/agent-skills
- Category
- AI & Agent Building
- First seen
- May 16, 2026
More from launchdarkly/agent-skills
All 41- Aiconfig Targeting2.2K
- Aiconfig Ai Metrics2K
- Aiconfig Custom Metrics2K
- Aiconfig Migrate2K
- Apply2K
- Detect2K
- First Flag2K
- Launchdarkly Experiment Setup2K
- Launchdarkly Guarded Rollout2K
- Mcp Configure2K
- Plan2K
- Sdk Install2K
- Aiconfig Agent Graphs1.8K
- Aiconfig Snippets1.8K
- Agent Graphs1.3K
- Built In Metrics1.3K
- Configs Create1.3K
- Configs Targeting1.3K
- Configs Update1.3K
- Configs Variations1.3K
- Custom Metrics1.3K
- Migrate1.3K
- Online Evals1.3K
- Projects1.3K
Recommended
More AI & Agent Building
mapbox-data-visualization-patternsmapbox/mapbox-agent-skillsPatterns for visualizing data on maps including choropleth maps, heat maps, 3D visualizations, data-driven styling, and animated data. Covers layer types, color2.2Kskills.sh installs
citation-managementlingzhi227/agent-research-skillsManage BibTeX citations for LaTeX papers. Harvest missing citations from a draft using Semantic Scholar, validate cite keys against .bib files, deduplicate entr2.2Kskills.sh installs
latex-postersk-dense-ai/scientific-agent-skillsCreate professional research posters in LaTeX using beamerposter, tikzposter, or baposter. Support for conference presentations, academic posters, and scientifi2.2Kskills.sh installs
jenkins-pipelineaj-geddes/useful-ai-promptsBuild Jenkins declarative and scripted pipelines with stages, agents, parameters, and plugins. Implement multi-branch pipelines and deployment automation.2.2Kskills.sh installspyzoterok-dense-ai/scientific-agent-skillsInteract with Zotero reference management libraries using the pyzotero Python client. Retrieve, create, update, and delete items, collections, tags, and attachm2.1Kskills.sh installsgenerate-imagek-dense-ai/scientific-agent-skillsGenerate or edit images with AI models through the OpenRouter Image API (Gemini, Seedream, Recraft, GPT-Image, Riverflow). Use for photos, illustrations, artwor2.1Kskills.sh installs