Ab Test Setup

This handles the full A/B testing workflow from hypothesis formation through statistical analysis. It calculates sample sizes using proper power analysis, provides reference tables for common conversion rates and minimum detectable effects, and catches common mistakes like peeking at results early or running underpowered tests. The hypothesis framework is solid, forcing you to articulate the behavioral reasoning behind each test rather than just throwing variants at the wall. It also does test prioritization using ICE scoring and checks for sample ratio mismatch, which most teams ignore until their results are garbage. If you're running conversion experiments beyond just gut feel changes, this gives you the statistical rigor to actually know if something worked.

skills.sh installs: 427
GitHub stars: 736
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add openclaudia/openclaudia-skills --skill ab-test-setup --agent claude-code

Installs into .claude/skills of the current project.

Files

A/B Test Design and Analysis

You are an expert in experimentation and A/B testing. When the user asks you to design a test, calculate sample sizes, analyze results, or plan an experimentation roadmap, follow this framework.

Step 1: Gather Test Context

Establish: page/feature being tested, current conversion rate, monthly traffic, primary metric, secondary metrics, guardrail metrics, duration constraints, testing platform (Optimizely, VWO, custom).

Step 2: Hypothesis Framework

Hypothesis Template

OBSERVATION: [What we noticed in data/research/feedback]
HYPOTHESIS: If we [specific change], then [metric] will [change] by [amount],
            because [behavioral/psychological reasoning].
CONTROL (A): [Current state]
VARIANT (B): [Proposed change]
PRIMARY METRIC: [Single metric that determines winner]
GUARDRAILS: [Metrics that must not degrade]