Langsmith Dataset

This is your toolkit for building evaluation datasets in LangSmith without writing a bunch of SDK boilerplate. It wraps the langsmith CLI so you can export traces, shape them into datasets (final response, single step, trajectory, or RAG formats), and upload them for testing. The workflow is straightforward: pull traces from a project, transform the JSON into examples with inputs and outputs, then push it back up. Handy if you're iterating on agent behavior and need regression tests or comparative evals. The CLI handles confirmations on destructive ops, which is nice when you're moving fast and don't want to accidentally nuke a dataset.

skills.sh installs: 4.9K
GitHub stars: 166
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add langchain-ai/langsmith-skills --skill langsmith-dataset --agent claude-code

Installs into .claude/skills of the current project.

Files

Create, manage, and upload evaluation datasets to LangSmith for testing and validation. Environment Variables

LANGSMITH_API_KEY=lsv2_pt_your_api_key_here          # REQUIRED
LANGSMITH_PROJECT=your-project-name                   # Check this to know which project has traces
LANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys

Authentication is REQUIRED. For agents, set the LANGSMITH_API_KEY environment variable (preferred); every langsmith command reads it automatically. To pass a key for a single command, use --api-key:

langsmith datasets list --api-key "$LANGSMITH_API_KEY"

IMPORTANT: Always check the environment variables or .env file for LANGSMITH_PROJECT before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one.

Python Dependencies