Content Refinement Agent

This is the fifth stage of the PaperOrchestra pipeline, where an initial draft gets iteratively refined through simulated peer review. It runs about three rounds of review-then-revise, checking each iteration against a six-axis rubric (scientific depth, technical execution, logical flow, writing clarity, evidence presentation, academic style). The score delta determines whether to accept the revision or revert to the previous snapshot, and the loop halts on revert or when no actionable weaknesses remain. The paper claims refinement alone added 19-22% to acceptance rates in their experiments. You get snapshots of every iteration, a worklog of decisions, and gates that check for hallucinated claims against your experimental log. Costs around five to seven LLM calls per run, mostly on the reviewer and revision prompts.

GitHub stars: 572
last updated: Jun 2, 2026

View on GitHub

Install

$ npx -y skills add ar9av/paperorchestra --skill content-refinement-agent --agent claude-code

Installs into .claude/skills of the current project.

Files

Content Refinement Agent (Step 5)

Faithful implementation of the Content Refinement Agent from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §4 Step 5, App. F.1 pp. 49–51).

Cost: ~5–7 LLM calls (App. B), typically ~3 refinement iterations, each consisting of one reviewer call and one revision call.

The paper highlights this step as one of the largest contributors to overall quality: refinement alone accounts for +19% (CVPR) and +22% (ICLR) absolute acceptance-rate improvement (Fig. 4). Get this step right.

Inputs

workspace/drafts/paper.tex — output of Step 4
workspace/inputs/conference_guidelines.md
workspace/inputs/experimental_log.md — used as ground truth for the hallucination check
workspace/citation_pool.json / workspace/refs.bib — the allowed bibliography

Outputs

workspace/refinement/iter1/, iter2/, iter3/ — per-iteration snapshots containing paper.tex, paper.pdf, review.json, score.json
workspace/refinement/worklog.json — append-only history of decisions
workspace/final/paper.tex and workspace/final/paper.pdf — copy of the best accepted snapshot