Claude MarketplacesSubmit
ar9av avatar

Paper Autoraters

ar9av/paperorchestra

Four LLM-as-judge metrics from the PaperOrchestra paper (arXiv:2604.05018) for scoring research paper quality. You get Citation F1 with P0/P1 partitioning, a six-axis literature review scorer with anti-inflation caps, and two side-by-side comparators (full paper and lit review only). The literature review rubric is strict by design: default range is 45-70, anything above 85 needs strong evidence on all axes, and there are hard penalties for overclaiming or citation dumping. Useful if you're building paper-writing agents and want the same benchmarks the paper used to compare against AI-Scientist-v2, or if you just want a systematic way to score drafts without handwaving.

GitHub stars
572
last updated
Jun 2, 2026
View on GitHub

Install

$ npx -y skills add ar9av/paperorchestra --skill paper-autoraters --agent claude-code

Installs into .claude/skills of the current project.

Files

Paper Autoraters (App. F.3)

Faithful implementation of the four LLM-as-judge autoraters used in PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).

These are the metrics the paper uses to demonstrate that PaperOrchestra beats single-agent and AI-Scientist-v2 baselines. Use them to:

  1. Score a generated paper against a ground-truth paper.
  2. Compare two paper-writing pipelines side-by-side.
  3. Validate your own host-agent execution of the paper-orchestra pipeline.

The four autoraters

SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Details

Repository
github.com/ar9av/paperorchestra
Categories
Git & Pull Requests, Code Review & Quality, Data Science & ML
First seen
Jun 11, 2026

More from ar9av/paperorchestra

All 9

Recommended

More Git & Pull Requests
1ar9av avatarpaper-orchestraar9av/paperorchestraOrchestrate the full PaperOrchestra (Song et al., 2026, arXiv:2604.05018) five-agent pipeline to turn unstructured research materials (idea, experimental log, L0skills.sh installs
2ar9av avatarplotting-agentar9av/paperorchestraStep 2 of the PaperOrchestra pipeline (arXiv:2604.05018). Execute the visualization plan from outline.json — render plots and conceptual diagrams from experimen0skills.sh installs
3artsy avatarbeepartsy/forcePlay a system beep. Use when asked to beep, play a beep, issue a system beep etc.0skills.sh installs
4awesome-skills avatarMobile App Design Standardsawesome-skills/mobile-app-designThis skill should be used when the user asks to "design mobile UI", "review app design", "check UI guidelines", "improve app UX", "design React Native interface0skills.sh installs
5awslabs avatarhyperpod-performance-debuggerawslabs/agent-pluginsDiagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-0skills.sh installs
6binjuhor avatarui-ux-pro-maxbinjuhor/shadcn-larFrontend UI/UX design intelligence - activate FIRST when user requests beautiful, stunning, gorgeous, or aesthetic interfaces. The primary skill for design deci0skills.sh installs