Agent Eval

Runs controlled benchmarks to measure whether CodeGraph actually helps Claude answer questions about a codebase compared to just using grep and file reads. You pick a repo from the corpus, choose between the local dev build or a published npm version, then kick off either headless runs for exact token counts or tmux sessions that mirror real TUI behavior. Takes several minutes since it clones repos, wipes indexes, and runs the agent twice per test. The output shows tool call counts, file reads, and total cost for each arm. Useful when you're validating changes to CodeGraph itself or need numbers to prove retrieval improvements on real codebases rather than synthetic tests.

skills.sh installs: 1.8K
GitHub stars: 77.2K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add colbymchenry/codegraph --skill agent-eval --agent claude-code

Installs into .claude/skills of the current project.

Files

CodeGraph Quality Audit

Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen codegraph version on a chosen real-world repo. Drives the harness in scripts/agent-eval/.

Prerequisites

tmux 3+, a logged-in claude CLI, node, git (macOS/Linux).
Run from the codegraph repo root.

Workflow

Copy this checklist:

- [ ] 1. Pick version (local or npm)
- [ ] 2. Pick language
- [ ] 3. Pick repo by size
- [ ] 4. Pick harness (headless / tmux / both)
- [ ] 5. Run audit.sh in the background
- [ ] 6. Report results

Step 1 — version. Ask with AskUserQuestion: which codegraph version to test. Offer "Local dev build" and "Latest published"; the free-text "Other" lets the user type a specific version (e.g. 0.7.10). Map the answer to a VERSION token:

"Local dev build" → local
"Latest published" → latest
a typed version → that string (e.g. 0.7.10)

Step 2 — language. Read .claude/skills/agent-eval/corpus.json. Ask with AskUserQuestion which language to test, listing the languages that have entries.

SKILL.md on GitHub