Dspy Evaluation Harness

You need this the moment you start wondering why your DSPy optimizer isn't improving. The real insight here is that GEPA lives or dies on rich textual feedback, not just scores. Returns a `dspy.Prediction(score=..., feedback=...)` instead of a dict (which breaks parallel aggregation), shows you how to build multi-axis metrics that actually teach the optimizer something useful, and hammers home the separate valset rule because optimizers overfit frighteningly fast. The canonical example walks through correctness plus citation plus conciseness with specific feedback strings. Also covers CI integration with cached LMs and the usual tracing hooks for MLflow and Weights & Biases. If your metric is a bare float, you're leaving signal on the table.

GitHub stars: 250
last updated: May 25, 2026

View on GitHub

Install

$ npx -y skills add intertwine/dspy-agent-skills --skill dspy-evaluation-harness --agent claude-code

Installs into .claude/skills of the current project.

Files

DSPy Evaluation Harness (3.2.x)

The metric is usually more important than the program. For dspy.GEPA especially, the quality of textual feedback in your metric determines whether optimization converges.

Two rules

Return a dspy.Prediction(score=..., feedback=...), not a dict. dspy.Evaluate's parallel executor aggregates scores via sum, which breaks on dict outputs (TypeError: unsupported operand type(s) for +: 'int' and 'dict'). dspy.Prediction supports __float__/__add__ and is what GEPA's adapter natively unwraps. A bare float still works for pure dspy.Evaluate scoring, but GEPA needs the score+feedback pair.
Separate valset. Never optimize and evaluate on the same examples. Optimizers overfit fast.