Ab Test Setup
This walks you through proper A/B test design from hypothesis to analysis. It pushes you to write structured hypotheses (the "Because X, we believe Y will cause Z" format), calculate sample sizes before you start, and pick one primary metric instead of getting lost in dashboards. The statistical rigor section is solid on common mistakes like peeking at results early or testing multiple things at once. Includes sample size tables, traffic allocation strategies, and the difference between client-side and server-side testing. It's opinionated about doing this right rather than just shipping tests fast, which is probably what you need if you keep launching experiments that run forever without reaching significance.
- skills.sh installs
- 356
- GitHub stars
- 33.2K
- last updated
- Aug 21, 2026
Install
$ npx -y skills add davila7/claude-code-templates --skill ab-test-setup --agent claude-codeInstalls into .claude/skills of the current project.
Files
A/B Test Setup
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Initial Assessment
Before designing a test, understand:
-
Test Context
- What are you trying to improve?
- What change are you considering?
- What made you want to test this?
-
Current State
- Baseline conversion rate?
- Current traffic volume?
- Any historical test data?
-
Constraints
- Technical implementation complexity?
- Timeline requirements?
- Tools available?
Core Principles
1. Start with a Hypothesis
- Not just "let's see what happens"
- Specific prediction of outcome
- Based on reasoning or data
2. Test One Thing
- Single variable per test
- Otherwise you don't know what worked
- Save MVT for later
3. Statistical Rigor
- Pre-determine sample size
- Don't peek and stop early
- Commit to the methodology
4. Measure What Matters
- Primary metric tied to business value
- Secondary metrics for context
- Guardrail metrics to prevent harm
Hypothesis Framework
Structure
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/davila7/claude-code-templates
- Categories
- Testing & QA, Data Science & ML, Productivity & Planning, Marketing & SEO, Design & UI/UX
- First seen
- Jun 3, 2026
More from davila7/claude-code-templates
All 618- Codex356
- Javascript Mastery356
- Model Merging356
- Artifacts Builder355
- Blip 2 Vision Language355
- Programmatic Seo355
- Biopython354
- Production Code Audit354
- Model Pruning353
- Fda Consultant Specialist352
- Voice Ai Development351
- Evaluating Code Models350
- Geo Database350
- Notion Spec To Implementation350
- Training Llms Megatron350
- Changelog Generator349
- Performance Profiling349
- Server Management349
- Tensorrt Llm349
- Deslop348
- Algorithmic Art347
- Email Systems347
- Faiss347
- Voice Agents347
Recommended
More Testing & QA
java-testing-advancedpluginagentmarketplace/custom-plugin-javaAdvanced testing - Testcontainers, contract testing, mutation testing, property-based331skills.sh installs
playwright-mcp-devmicrosoft/playwrightExplains how to add and debug playwright MCP tools and CLI commands.313skills.sh installs
alicloud-ai-multimodal-qwen-vl-testcinience/alicloud-skillsMinimal image-understanding smoke test for Model Studio Qwen VL.301skills.sh installsalicloud-ai-video-wan-r2v-testcinience/alicloud-skillsMinimal reference-to-video smoke test for Model Studio Wan R2V.299skills.sh installsalicloud-ai-entry-modelstudio-matrix-testcinience/alicloud-skillsMinimal matrix test wrapper for alicloud-ai-entry-modelstudio-test.297skills.sh installsalicloud-data-lake-dlf-next-testcinience/alicloud-skillsMinimal smoke test for DlfNext skill. Validate metadata discovery and one read-only API call.297skills.sh installs