Ab Test Setup

This is a structured checklist that stops you from running A/B tests before you've actually thought them through. It enforces hard gates at hypothesis definition, sample size calculation, and metric selection, refusing to proceed until you've locked in a primary metric and calculated whether you have enough traffic to detect your minimum effect size. The discipline here is real: no peeking at results early, no changing success criteria mid-test, and guardrail metrics that can veto a win if they tank. Use it when you need to run statistically valid experiments and want something that will actively push back on sloppy test design. It's opinionated about rigor over speed, which is either exactly what you need or going to feel heavy handed depending on your team's maturity.

skills.sh installs: 736
GitHub stars: 43.1K
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add sickn33/antigravity-awesome-skills --skill ab-test-setup --agent claude-code

Installs into .claude/skills of the current project.

Files

A/B Test Setup

1️⃣ Purpose & Scope

Define an experiment that can answer a specific product question, and verify its assumptions before exposing users. This procedure cannot guarantee validity by itself.

Documents the stopping rule
Estimates sample needs under stated assumptions
Makes the hypothesis and decision criteria reviewable

2️⃣ Pre-Requisites

You must have:

A clear user problem
Access to an analytics source
Roughly estimated traffic volume

Hypothesis Quality Checklist

A valid hypothesis includes:

Observation or evidence
Single, specific change
Directional expectation
Defined audience
Measurable success criteria

3️⃣ Hypothesis Lock (Hard Gate)

Before designing variants or metrics, you MUST:

Present the final hypothesis
Specify:
- Target audience
- Primary metric
- Expected direction of effect
- Minimum Detectable Effect (MDE)

Use the hypothesis already agreed in the task. If a launch-critical choice is missing, present the concrete choice for confirmation while continuing independent analysis. Do not repeatedly request approval for a decision already authorized.

SKILL.md on GitHub