Calculator

A/B Testing Power & Sample Size Calculator (Interactive Sample Plan)

An interactive sample-plan builder that helps teams specify a defensible A/B test: enter baseline rates or means, select MDE, power, alpha, allocation, and traffic. Saves a reproducible plan and explains calculations, common pitfalls (peeking, multiple comparisons, sequential testing), and how to estimate test duration. Designed to store plans and integrate with a calculation engine or CSV export.

Interactive Tool

A/B Testing Power & Sample Size Calculator

Design tests that can actually answer your question

This tool helps you build a clear, reproducible sample-size and timing plan for common A/B experiments. Enter your baseline (conversion rate or mean and SD), choose a minimum detectable effect (MDE), and set statistical choices such as power, significance, sidedness, allocation and traffic. The form saves a complete test plan you can share, export, or run through an automatic calculator when your team enables the calculation engine.

Why this matters: Underpowered tests or vague MDEs lead to wasted time and misleading null results. Being explicit about assumptions (baseline, MDE, allocation) helps you interpret outcomes and avoid quietly post‑hoc rationalization.

Quick guidance and formulas

For two-sample proportions (equal allocation) a commonly used approximate formula for the sample size per group is:

n ≈ [ (Z_{1-α/2} * sqrt(2 * p̄ * (1 - p̄)) + Z_{power} * sqrt(p1*(1-p1) + p2*(1-p2)))^2 ] / (p1 - p2)^2

Where p1 is baseline, p2 = p1 ± MDE, and p̄ = (p1 + p2)/2. For continuous outcomes (means) you need an estimate of the standard deviation and can use the familiar two-sample t-test approximations.

The tool supports both conversion-rate (proportion) and continuous-mean scenarios, along with options for absolute/relative MDE and one- or two-sided tests.

Practical cautions

  • Peeking & sequential testing: Repeatedly checking results inflates false positives. Pre-specify stopping rules or use sequential methods and appropriate corrections.
  • Multiple comparisons: If you test many variants, adjust significance (e.g., Bonferroni, BH) or use multi-arm methods to avoid excess Type I error.
  • External validity: Ensure the experiment environment reflects the population you want to generalize to; implementation differences can invalidate results.

Using this form

Fill required fields and press Calculate & Save Plan. The platform will store the plan for your team. If the calculation engine is available, the form will also compute sample sizes and estimate duration from traffic inputs; otherwise use the guidance and formula above or export the inputs as CSV for an offline calculation.

A short descriptive name for this experiment (e.g., 'Homepage CTA color test').
Choose whether the primary metric is a conversion (proportion) or a continuous metric (mean).
Enter baseline conversion as a percentage (e.g., 4.5 for 4.5%). Required for proportion tests.
Enter the baseline mean value for continuous outcomes (e.g., average order value).
Estimate of the standard deviation for the continuous metric. Required for continuous tests.
Choose whether your Minimum Detectable Effect is an absolute difference or a relative percent change.
If relative, enter the percent change you care about (e.g., 10 for 10%). If absolute and using conversions, enter percentage points (e.g., 1.2 for +1.2%).
Common values: 0.05. Use smaller alpha for stricter control of false positives or when doing many tests.
Probability of detecting the MDE if it truly exists. Common values: 0.8 or 0.9.
Choose one- or two-sided test. Two-sided is default unless you have a strong directional hypothesis.
For a simple A/B test enter 2. For multi-arm tests enter the total number of arms. Multi-arm calculations are approximate here.
Relative allocation per variant. For equal allocation in a 2-arm test use 1 (control:variant). For 50/50 split between two arms use 1. To allocate more traffic to control use >1.
Helps estimate calendar duration. Enter the total number of test-eligible visitors per day (not per arm).
Whether you expect an increase or decrease. Useful for interpreting relative MDE inputs.
Any implementation constraints, segments, or seasonal concerns to record in the plan.
Choose yes to save the plan and prepare a CSV of the input fields for offline calculation or sharing.
You can explore this tool now. Sign in or create an account to save your responses and return to them later.
Sign In to Save Create an Account
Make this tool part of your work

Save a personal copy, bring it to your team, or tailor the questions and workflow to fit what you are hungry to improve.

Member customization and team collaboration are coming soon.
Discuss Ask the Librarian

Discussion

Comments and conversation will live here.