Trustworthy Experiments

pmprompt/claude-plugin-product-management

This is Ronny Kohavi's framework for running A/B tests that actually produce trustworthy results. It helps you design experiments with proper sample sizes, validate results before shipping, and avoid the false positives that plague most testing programs. The core insight is sobering: 66-92% of experiments fail to improve metrics, and with an 8% base success rate, a p-value of 0.05 still means 26% false positive risk. Use this when you're setting up experimentation infrastructure, diagnosing suspicious results, or need to make confident ship/no-ship decisions. Skip it if you don't have tens of thousands of users or need immediate answers.

skills.sh installs: 353
GitHub stars: 58
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add pmprompt/claude-plugin-product-management --skill trustworthy-experiments --agent claude-code

Installs into .claude/skills of the current project.

Files

Domain Context

This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work.

Input Requirements

Context about your product, feature, or problem
Relevant data, research, or constraints (recommended but optional)
Clear articulation of what you're trying to achieve

Trustworthy Experiments

What It Is

Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.

The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"

Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:

66-92% of experiments fail to improve the target metric
8% of experiments have invalid results due to sample ratio mismatch alone
When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk