Promptfoo Evaluation

Sets up and runs LLM evaluations with Promptfoo, an open-source CLI for testing prompts across different models. You'll reach for this when you need to compare Claude and GPT outputs side by side, write custom Python assertions for specific quality checks, or use LLM-as-judge scoring with rubrics. The skill covers the whole workflow: creating promptfooconfig.yaml, managing test cases with variable injection, implementing few-shot examples in chat format, and handling the gotchas like maxConcurrency placement and file path resolution. One thing to watch: if you're running through a relay API, every llm-rubric assertion needs its own provider config with apiBaseUrl or you'll hit 401 errors.

skills.sh installs: 1.6K
GitHub stars: 1.5K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add daymade/claude-code-skills --skill promptfoo-evaluation --agent claude-code

Installs into .claude/skills of the current project.

Files

Promptfoo Evaluation

Overview

This skill provides guidance for configuring and running LLM evaluations using Promptfoo, an open-source CLI tool for testing and comparing LLM outputs.

Quick Start

# Initialize a new evaluation project
npx promptfoo@latest init

# Run evaluation
npx promptfoo@latest eval

# View results in browser
npx promptfoo@latest view

Configuration Structure

A typical Promptfoo project structure:

project/
├── promptfooconfig.yaml    # Main configuration
├── prompts/
│   ├── system.md           # System prompt
│   └── chat.json           # Chat format prompt
├── tests/
│   └── cases.yaml          # Test cases
└── scripts/
    └── metrics.py          # Custom Python assertions

Core Configuration (promptfooconfig.yaml)

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: "My LLM Evaluation"

# Prompts to test
prompts:
  - file://prompts/system.md
  - file://prompts/chat.json

# Models to compare
providers:
  - id: anthropic:messages:claude-sonnet-4-6
    label: Claude-Sonnet-4.6
  - id: openai:gpt-4.1
    label: GPT-4.1

SKILL.md on GitHub