Paper Writing Bench

This replicates the dataset construction procedure from PaperOrchestra (arXiv:2604.05018) so you can build your own benchmark cases for testing paper-writing pipelines. Feed it a research paper PDF and it reverse-engineers three files: a sparse idea (high-level, no math), a dense idea (full LaTeX equations), and an experimental log (raw numbers, stripped of all narrative). The prompts are taken verbatim from Appendix F.2 and enforce strict rules like no citation leakage, no figure references in the log, and complete anonymization. Use this when you want to test whether your pipeline can reconstruct a real paper from just the raw materials, then score the output with paper-autoraters against the original.

GitHub stars: 572
last updated: Jun 2, 2026

View on GitHub

Install

$ npx -y skills add ar9av/paperorchestra --skill paper-writing-bench --agent claude-code

Installs into .claude/skills of the current project.

Files

PaperWritingBench (§3)

Faithful implementation of the PaperWritingBench dataset construction procedure from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §3 and App. C, F.2).

The original benchmark contains 200 papers (100 CVPR 2025 + 100 ICLR 2025). For each paper, the authors reverse-engineer the (I, E) tuple by stripping narrative flow from the original PDF using the three prompts in App. F.2. You can use this skill to reverse-engineer your own benchmark cases from any paper PDF.

What this skill does

Given an existing AI research paper (PDF or markdown extract), produce:

idea.md (Sparse variant) — high-level concept note, no math, no experimental results
idea.md (Dense variant) — detailed technical proposal with LaTeX equations and variable definitions, but still no experimental results
experimental_log.md — exhaustive raw experimental setup, numeric data, and qualitative observations, with all narrative references stripped

These three files form a complete (I, E) input pair for the paper-orchestra pipeline. You can then run the pipeline and compare its output to the original paper using paper-autoraters.