Nemo Evaluator Sdk

NVIDIA's enterprise benchmarking platform that runs your LLMs through 100+ evaluation tasks from 18+ harnesses including MMLU, HumanEval, and GSM8K. Works with any OpenAI-compatible endpoint and handles execution across local Docker, Slurm HPC clusters, or cloud platforms. The containerized approach means reproducible results, and you get built-in exports to MLflow and Weights & Biases. If you're running evals on a single machine with simpler needs, lm-evaluation-harness is lighter weight. But if you're benchmarking at scale across infrastructure or need that full harness coverage in one tool, this delivers the industrial-grade setup.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill nemo-evaluator-sdk --agent claude-code

Installs into .claude/skills of the current project.

Files

NeMo Evaluator SDK - Enterprise LLM Benchmarking

Quick Start

NeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).

Installation:

pip install nemo-evaluator-launcher

Set API key and run evaluation:

export NGC_API_KEY=nvapi-your-key-here

# Create minimal config
cat > config.yaml << 'EOF'
defaults:
  - execution: local
  - deployment: none
  - _self_

execution:
  output_dir: ./results

target:
  api_endpoint:
    model_id: meta/llama-3.1-8b-instruct
    url: https://integrate.api.nvidia.com/v1/chat/completions
    api_key_name: NGC_API_KEY

evaluation:
  tasks:
    - name: ifeval
EOF

# Run evaluation
nemo-evaluator-launcher run --config-dir . --config-name config

View available tasks: