Experiment & A/B Test Template (Design of Experiments)

Purpose

Use this template to design, document, run, analyze, and hand off experiments and A/B tests so results are credible, reproducible, and safe to act on. Fill each section before starting data collection and pre-register the Analysis Plan and Stopping Rules.

Experiment Title

[Short descriptive name]

Experiment ID

[Canonical ID used in analytics and tracking]

Owner(s) & Stakeholders

• Experiment Owner: [name, role]
• Data/Analytics Lead: [name]
• Engineering/Instrumentation Lead: [name]
• Product/Business Sponsor: [name]
• Safety/Compliance Contact (if applicable): [name]

1. Hypothesis & Success Metric

State a clear, testable hypothesis and identify one primary success metric. Avoid vague language.

• Hypothesis (If-then): If we [change X], then [desired behavior] will [increase/decrease] by [expected magnitude or direction].
• Primary Metric (one): [Name of metric, definition, numerator/denominator, unit, desired direction].
• Secondary Metrics: [List additional metrics and why they matter].
• Guardrail Metrics: [Metrics to monitor for unintended harm].

2. Population & Randomization

• Population: Inclusion and exclusion criteria (geography, device, user segment, traffic source, account type).
• Unit of Randomization: (e.g., user, session, account, cookie, device).
• Randomization Method: ID hashing, Rand() in experiment platform, stratified randomization? Describe mechanism and seed control for reproducibility.
• Assignment Ratio: (e.g., 1:1, 2:1). Justify any skewed allocation.
• Start/End Time Window: Planned calendar dates and business season considerations.

3. Sample Size Calculation

Explain expected baseline, minimum detectable effect (MDE), desired power and significance level. Record the calculation and the tool used.

• Baseline rate/value: [value and measurement window]
• Minimum detectable effect: [absolute or relative change you want to detect]
• Alpha (Type I error): typically 0.05
• Power (1 - Type II error): typically 0.8 or 0.9
• One-sided or Two-sided test: state which and why
• Calculated sample size: [n per variant] — include the calculator URL, inputs, and timestamp of calculation

Quick power calculator: [Insert working power-calculator link here]

4. Data Sources & Instrumentation Checklist

List each data source, event, and property required to compute primary, secondary, and guardrail metrics. Confirm ownership and test instrumentation before running.

• Event & property names (exact): [list]
• Where data is stored: [analytics tool, DB, event stream]
• Telemetry sampling: [none / fraction; if sampled, describe plan to correct]
• Required joins or derived fields: [description]
• Data freshness/latency and when analysis is valid
• QA steps to validate instrumentation:
• Smoke test: verify events appear in analytics within X minutes
• Sanity check: confirm split proportions in raw event stream match intended allocation
• Metric parity: compare key metrics vs. baseline system for a short pilot window
• Edge cases: logged out users, bots, and internal traffic exclusion

5. Analysis Plan (Pre-registered)

Pre-register the full analysis. Include exact definitions, transformations, statistical tests, and handling of missing data.

• Primary analysis: estimator (e.g., difference in proportions, t-test on means, regression adjustment), confidence intervals, one/two-sided.
• Adjustment covariates: any pre-specified covariate adjustments (e.g., baseline metrics, device, geography) and modeling approach.
• Secondary analyses: list and order (exploratory vs confirmatory).
• Multiple comparisons: correction method if testing multiple hypotheses (Bonferroni, FDR, hierarchical testing).
• Subgroup analyses: pre-specified subgroups and criteria for interpretation.
• Data cleaning rules: outlier handling, exclusion criteria, minimum exposure time/window.
• Missing data: expected types and handling strategy.
• Reproducible analysis: analysis notebook location, code repo, and dataset snapshot ID.

6. Stopping Rules

Define explicit stopping criteria to avoid ad-hoc peeking and inflated false positives.

• Planned duration and minimum sample size per arm before any look.
• Interim analyses: allowed frequency and statistical method (alpha spending, group sequential design) or none.
• Futility rules: criteria to stop early for lack of effect.
• Safety rules: immediate stop if guardrail metric crosses safety threshold.
• Decision authority: who may stop the experiment and under what governance process.

7. Post-Test Actions & Decision Rules

Explicit decision rules link results to actions. Do not leave interpretation to informal judgment.

• If primary metric shows statistically significant and practically meaningful improvement and no guardrail violations → proceed to staged rollout (describe rollout plan).
• If no significant effect and not harmful → declare null and optionally iterate on hypotheses.
• If harmful on guardrails → stop and roll back immediately; perform root-cause analysis.
• Document all decisions and rationale in the experiment record.

8. Rollout Criteria & Rollback Plan

• Rollout strategy: canary → gradual % increases → full deployment. Specify cadence and monitoring windows.
• Monitoring during rollout: dashboards, alert thresholds, runbook contacts.
• Rollback triggers: metric thresholds, incident reports, performance regressions.
• Rollback procedure: step-by-step including owner, expected downtime, and communication plan.

9. Handoff to Production & Documentation

• Final implementation artifact location (repo, PR, release)
• Production owner(s) and support contacts
• Updated runbooks and monitoring dashboards
• What to include in the changelog and customer communications (if any)
• Archive experiment materials (data snapshots, analysis notebooks, decision log) and link to the experiment record

10. Ethics, Privacy & Compliance

• Identify any privacy risks and required consents.
• Data retention and anonymization plan.
• Regulatory considerations (e.g., medical, financial, minors).
• Any required IRB or legal review completed? [yes/no/details]

Worked example (illustrative, fill-in)

Fill the template fields with concrete values before launch. Below is a short illustrative example that shows how the template is used; this is not a full analytical output.

• Hypothesis: If we simplify the checkout form (remove optional fields), then checkout conversion (primary metric: completed purchases / checkout sessions) will increase.
• Primary metric: 7-day purchase conversion rate per unique user.
• Population: New and returning users on desktop in US, excluding internal traffic.
• MDE: detect an absolute lift of 2 percentage points vs baseline ~10%.
• Alpha / Power: 0.05 two-sided / power 0.8.
• Sample size: [Use power-calculator link to compute; paste calculator result and timestamp here.]
  Example: "Power calculator used: [link]. Inputs: baseline=10%, lift=2pp, alpha=0.05, power=0.8 → sample per arm = [paste result]."
• Analysis plan: difference in proportions with covariate adjustment for device and referral source; report 95% CI; pre-specified subgroup by new vs returning users.
• Stopping rules: do not look before minimum sample is reached; one interim at 50% sample using O'Brien-Fleming alpha-spend if needed.

Appendix: Quick Pre-Launch QA Checklist

• Experiment code deployed to staging and passes integration tests
• Instrumentation smoke-tested and events appear in analytics
• Split ratios verified in event logs
• Data retention/access permissions set
• Stakeholders briefed and decision owners assigned
• Runbook and alerting created for rollout phase

Links & Resources

• Power calculator: [insert link]
• Experiment registry: [link to central registry]
• Analysis notebook template: [link to repo/notebook]
• Instrumentation guidelines: [link]