Claude MarketplacesSubmit
phuryn avatar

Ab Test Analysis

phuryn/pm-skills

This handles the full statistical analysis workflow for A/B tests: validates sample sizes and test duration, calculates significance with p-values and confidence intervals, checks for sample ratio mismatches, and gives you a clear ship/extend/stop recommendation. It can read CSV or Excel files directly and generates Python scripts for the calculations. The best part is the decision framework that accounts for guardrail metrics, so you won't ship a win that tanks revenue or engagement elsewhere. It walks through hypothesis validation, effect size calculations, and business context in one pass. Useful when you need to move from raw experiment data to an actual product decision without second-guessing the stats.

skills.sh installs
3K
GitHub stars
27.8K
last updated
Aug 21, 2026
View on GitHub

Install

$ npx -y skills add phuryn/pm-skills --skill ab-test-analysis --agent claude-code

Installs into .claude/skills of the current project.

Files

A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

Context

You are analyzing A/B test results for $ARGUMENTS.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

Instructions

  1. Understand the experiment:

    • What was the hypothesis?
    • What was changed (the variant)?
    • What is the primary metric? Any guardrail metrics?
    • How long did the test run?
    • What is the traffic split?

  2. Validate the test setup:

    • Sample size: Is the sample large enough for the expected effect size?
      • Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
      • Flag if the test is underpowered (<80% power)
    • Duration: Did the test run for at least 1-2 full business cycles?
    • Randomization: Any evidence of sample ratio mismatch (SRM)?
    • Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Details

Repository
github.com/phuryn/pm-skills
Categories
Testing & QA, Release Management, Productivity & Planning
First seen
May 16, 2026

More from phuryn/pm-skills

All 68

Recommended

More Testing & QA
1eronred avatarab-test-store-listingeronred/aso-skillsWhen the user wants to A/B test App Store product page elements to improve conversion rate. Also use when the user mentions "A/B test", "product page optimizati2.5Kskills.sh installs
2owl-listener avatardesign-qa-checklistowl-listener/designer-skillsCreate QA checklists for verifying design implementation accuracy.2.5Kskills.sh installs
3owl-listener avatartest-scenarioowl-listener/designer-skillsGenerates structured usability test scenarios with realistic tasks, success criteria, and facilitation notes — ready to run with real participants or in a moder2.3Kskills.sh installs
4vinvcn avatarqavinvcn/mattpocock-skills-zh-cn交互式 QA 会话，用户以对话方式报告缺陷或问题，代理创建 GitHub issues。后台探索代码库以获取上下文和领域语言。适用于用户想报告缺陷、执行 QA、以对话方式提交 issues，或提到 “QA session” 时。2.1Kskills.sh installs
5anthropics avatarinitiating-coverageanthropics/financial-servicesCreate institutional-quality equity research initiation reports through a 5-task workflow. Tasks must be executed individually with verified prerequisites - (1)2Kskills.sh installs
6anthropics avatarcash-flow-snapshotanthropics/knowledge-work-pluginsReads AR/AP, historical cash timing, and known fixed costs from QuickBooks, PayPal, Stripe, or Square — or a CSV upload — and produces a 30/60/90-day cash flow 2Kskills.sh installs