Template

Design of Experiments One‑Pager (A/B & Causal Tests)

An interactive, saveable experiment-design template that captures hypothesis, primary/secondary metrics, power/sample notes, randomization, guardrails, analysis plan, decision rules, and reproducibility artifacts. Includes a brief worked example and internal readiness checklist.

Interactive Tool

Design of Experiments One‑Pager (A/B & Causal Tests)

Use this structured template to design reliable, reproducible causal tests. Capture the hypothesis, clear metrics, sample-size notes, guardrails, and an analysis plan that will produce actionable decisions. Save the design for internal review, reproducibility, and organizational learning.

Worked example (brief)

Business question: Will a simplified checkout reduce cart abandonment?
Hypothesis: Removing optional form fields will reduce abandonment by at least 10% relative to baseline.
Primary metric: 7-day cart-to-order conversion rate (numerator: orders within 7 days; denominator: carts started). Baseline: 8%. MDE: 10% relative (0.8 percentage points).
Analysis plan: Intention-to-treat A/B test; two-sided alpha = 0.05; power = 0.8; randomize by user account; pre-registered script; decision rule: p < 0.05 and effect >= MDE -> staged rollout to 50% then monitor for 2 weeks.

Business question

The high-level decision this experiment will inform (who cares and why).

Hypothesis

A concise, testable statement linking the change to an expected outcome (if... then...).

Primary metric name

The single metric you will use to decide success or failure. State the precise calculation (numerator/denominator).

Primary metric definition

Operational definition: exact calculation, filters, time windows, and data source.

Secondary metrics

Other metrics to monitor for signals, safety, or unintended consequences.

Baseline value (numeric)

Current value of the primary metric (use same units as metric). If unsure, estimate and note source.

Minimum detectable effect (MDE) — percent relative

Smallest relative change worth detecting (e.g., enter 10 for 10%). Helps estimate sample size.

Alpha (significance level)

Probability of Type I error. Common default is 0.05.

Power (1 - Type II error)

Desired probability of detecting the MDE if it exists. Common default is 0.8 (80%).

Estimated sample size per variant

If you have a sample-size estimate, enter it here. Otherwise use an external calculator and paste the result. Consider clustering or loss to follow-up in your estimate.

Planned segmentation and analysis groups

Which subgroups will be analyzed (e.g., new vs returning users, device type). Avoid excessive slicing unless powered.

Randomization unit

The entity assigned to variants (user, session, account, device, site, cluster, etc.). Choose the smallest independent unit appropriate.

Randomization plan and allocation

How randomization will be implemented, any stratification, allocation ratios, hashing method, and seed storage.

Implementation notes

Deployment plan, feature flags, monitoring endpoints, rollback mechanism, and responsible implementers.

Expected harms or negative outcomes

List potential adverse effects, accessibility, legal, privacy, or safety concerns.

Mitigation measures

Plans to detect and reduce harms (guardrails, kill switch, manual monitoring, notifications).

Analysis plan

Statistical tests, handling of missing data, stopping rules for peeking, covariates, and whether analysis is intention-to-treat or per-protocol. Specify exact tests and code locations if possible.

Analysis owner

Person or team responsible for analysis and reporting (name or role).

Estimated timeline

Start date, expected test duration, and checkpoints (dates or durations).

Decision rule (roll / stop)

Formal rule mapping analysis outcomes to actions (e.g., if p < 0.05 and effect >= MDE then stage rollout to 50%). Be specific about secondary checks and monitoring windows.

Planned rollout after positive result

Steps for gradual rollout, metrics to watch during scaling, and rollback thresholds.

Data, code and reproducibility plan

Where raw and processed data, analysis scripts, randomization seeds, and environment specs will be stored; access permissions and retention policy.

Consent and ethics

Any consent, privacy, or regulatory issues and approvals required for the experiment.

Reproducibility checklist

Essential reproducibility artifacts — check those you will provide.

Internal review checklist

Quick internal readiness review before execution.

Discussion

Comments and conversation will live here.

Design of Experiments One‑Pager (A/B & Causal Tests)