Langsmith Evaluator

This walks you through the three pieces you need for LangSmith evaluations: writing evaluators (LLM-as-judge or custom code), defining run functions that capture your agent's outputs, and actually running the evals either locally with evaluate() or by uploading them via the CLI. The golden rule here is solid: always inspect your actual output structure before writing extraction logic, because frameworks vary wildly. One thing to watch: LLM-as-judge evaluators can't be uploaded yet, only run locally, so you'll want to use evaluate() with local evaluators for dataset comparisons. The examples cover both Python and TypeScript, and there's a helpful table showing the differences between local and uploaded evaluator behavior, which matters more than you'd think for return formats.

skills.sh installs: 5.1K
GitHub stars: 166
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add langchain-ai/langsmith-skills --skill langsmith-evaluator --agent claude-code

Installs into .claude/skills of the current project.

Files

Three core components: **(1) Creating Evaluators** - LLM-as-Judge, custom code; **(2) Defining Run Functions** - capture agent outputs/trajectories for evaluation; **(3) Running Evaluations** - locally with `evaluate()` or auto-run via uploaded evaluators. Python and TypeScript examples included. Environment Variables

LANGSMITH_API_KEY=your-api-key                        # REQUIRED for agents (preferred auth)
LANGSMITH_ENDPOINT=https://api.smith.langchain.com   # SDK/CLI environment
LANGSMITH_PROJECT=your-project-name                   # Default for trace/run queries only
LANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys
OPENAI_API_KEY=your-openai-key                        # For locally executed OpenAI judges

SKILL.md on GitHub

Details

Repository: github.com/langchain-ai/langsmith-skills
Categories: AI & Agent Building, CLI & Terminal

More from langchain-ai/langsmith-skills

All 3

Langsmith Evaluator

Install

Files

Details

More from langchain-ai/langsmith-skills

Recommended