Auto Arena
Need to compare multiple LLMs or agents but don't have test data ready? This generates test queries from your task description, collects responses from all your endpoints, creates evaluation rubrics on the fly, then runs pairwise comparisons with position-bias swapping to produce win-rate rankings. It's built on OpenJudge's AutoArenaPipeline and handles the full workflow from query generation through final reports with charts. Works entirely via config YAML or Python API, supports checkpointing so you can resume interrupted runs, and lets you swap judge models mid-evaluation without rerunning everything. Solid choice when you want arena-style benchmarking without manually curating datasets or evaluation criteria first.
- GitHub stars
- 657
- last updated
- Jun 9, 2026
Install
$ npx -y skills add agentscope-ai/openjudge --skill auto-arena --agent claude-codeInstalls into .claude/skills of the current project.
Files
Need to compare multiple LLMs or agents but don't have test data ready? This generates test queries from your task description, collects responses from all your endpoints, creates evaluation rubrics on the fly, then runs pairwise comparisons with position-bias swapping to produce win-rate rankings. It's built on OpenJudge's AutoArenaPipeline and handles the full workflow from query generation through final reports with charts. Works entirely via config YAML or Python API, supports checkpointing so you can resume interrupted runs, and lets you swap judge models mid-evaluation without rerunning everything. Solid choice when you want arena-style benchmarking without manually curating datasets or evaluation criteria first.
Browse the repository on GitHubSponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/agentscope-ai/openjudge
- License
- Apache-2.0
- First seen
- Jun 11, 2026
More from agentscope-ai/openjudge
All 9Recommended
cavemanjuliusbrussee/cavemanUltra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman while keeping full technical accuracy. Supports intensity levels618.3Kskills.sh installs
improveshadcn/improveSurvey any codebase as a senior advisor and produce prioritized, self-contained implementation plans for OTHER models/agents to execute. Strictly read-only on s40Kskills.sh installs
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes326.7Kskills.sh installs
karpathy-guidelinesforrestchang/andrej-karpathy-skillsBehavioral guidelines to reduce common LLM coding mistakes through explicit assumptions, simplicity, and verifiable success criteria.17.9Kskills.sh installs
find-skillsvercel-labs/skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express int4.3Mskills.sh installs