Evaluating Llms Harness

This wraps the LM Evaluation Harness, the standard tool for benchmarking language models across 60+ academic datasets like MMLU, GSM8K, and HellaSwag. You point it at any HuggingFace model and get apples-to-apples comparisons using consistent prompts and metrics. It's what most researchers use when they publish benchmark numbers, so if you're fine-tuning models or need to justify which one to deploy, this gives you the same numbers everyone else is citing. The skill makes it available in Claude Code workflows, though you're mostly just getting a thin wrapper around lm_eval's CLI. Useful if you're already doing AI research work and want benchmarks without leaving your agent environment.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill evaluating-llms-harness --agent claude-code

Installs into .claude/skills of the current project.

Files

lm-evaluation-harness - LLM Benchmarking

Quick start

lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Installation:

pip install lm-eval

Evaluate any HuggingFace model:

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag \
  --device cuda:0 \
  --batch_size 8

View available tasks:

lm_eval --tasks list

Common workflows

Workflow 1: Standard benchmark evaluation

Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval).

Copy this checklist:

Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model
- [ ] Step 3: Run evaluation
- [ ] Step 4: Analyze results

Step 1: Choose benchmark suite

Core reasoning benchmarks:

MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice
GSM8K - Grade school math word problems
HellaSwag - Common sense reasoning
TruthfulQA - Truthfulness and factuality
ARC (AI2 Reasoning Challenge) - Science questions

SKILL.md on GitHub