Evaluating Code Models

This wraps the BigCode Evaluation Harness, which benchmarks code generation models across 15+ standardized tests including HumanEval, MBPP, and MultiPL-E supporting 18 programming languages. You'd reach for this when you need to compare model performance systematically rather than relying on vibes, or when you're fine-tuning a code model and want real numbers on whether your changes actually help. The harness is widely used in research (9.2K stars), so your benchmarks will be comparable to published results. Setup requires cloning the repo and running accelerate config, so it's not quite as plug-and-play as some skills, but the tradeoff is you get the full evaluation suite that serious ML teams use.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill evaluating-code-models --agent claude-code

Installs into .claude/skills of the current project.

Files

BigCode Evaluation Harness - Code Model Benchmarking

Quick Start

BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).

Installation:

git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
accelerate config

Evaluate on HumanEval:

accelerate launch main.py \
  --model bigcode/starcoder2-7b \
  --tasks humaneval \
  --max_length_generation 512 \
  --temperature 0.2 \
  --n_samples 20 \
  --batch_size 10 \
  --allow_code_execution \
  --save_generations

View available tasks:

python -c "from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)"

Common Workflows

Workflow 1: Standard Code Benchmark Evaluation

Evaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).

Checklist:

Code Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model and generation
- [ ] Step 3: Run evaluation with code execution
- [ ] Step 4: Analyze pass@k results

Step 1: Choose benchmark suite

SKILL.md on GitHub