Design of Experiments & A/B Testing Toolkit

Welcome

This toolkit helps teams run experiments that produce credible causal learning and clear decision rules. It focuses on practical recipes you can use today: hypothesis framing, metric selection, sample size & power, randomization, pragmatic analysis, and recorded decisions so learnings can be trusted and scaled.

What's included

• Sample-size & power formulas with worked examples
• Copy‑ready pre-registration template (paste & adapt)
• Analysis checklist to use before unblinding results
• Guardrails for multiple testing and sequential looks
• Post-experiment interpretation checklist
• Short primer on common threats to validity and mitigations
• Practical adoption notes for turning this into interactive forms and organizational templates

Quick start: A pragmatic 5-step approach

1. Write a one-sentence causal hypothesis (what change you expect and why).
2. Choose a primary metric aligned with business value and define it precisely.
3. Estimate effect size you care about and compute required sample size.
4. Pre-register the design and analysis plan; lock it before looking at results.
5. Run the experiment, follow the analysis checklist, interpret using the post-experiment checklist, and record decisions for reuse.

Sample-size & power (practical formulas)

Two formulas below are the usual starting points. Use them as conservative guides; consider variance estimates from historical data when available.

1) Two-sample test for proportions (baseline p, detect absolute lift d)

Approximate sample size per group:

n ≈ [ (Z_{1-α/2} √(2 p̄ (1-p̄)) + Z_{1-β} √(p1(1-p1)+p2(1-p2)) )² ] / d²

Where p̄=(p1+p2)/2, p1=baseline, p2=p1+d. For typical industry defaults: α=0.05 (Z=1.96), power=0.8 (Z=0.84).

Worked example: baseline p1=0.10, want to detect d=0.01 (1 percentage point). Plugging values gives an order-of-magnitude n ~ 40,000 per arm (illustrative — calculate precisely with a tool).

2) Two-sample test for means (continuous metric)

n per group ≈ [ (Z_{1-α/2}+Z_{1-β})² * 2 σ² ] / Δ²

Where σ² is variance (or pooled variance estimate) and Δ is the minimum detectable difference. If σ is unknown, estimate from historical weekly data.

Worked example: σ=10 units, Δ=1 unit, α=0.05, power=0.8 → n ≈ ((1.96+0.84)² * 2 * 100) / 1² ≈ 1,536 per arm.

Notes:

• For small expected effects on low-baseline rates, required sample sizes can grow large — consider redesigning the metric or experiment (e.g., use a lift-on-ratio, pooled metrics, or targeted segments).
• Clustered randomization (by user, account, or session) increases required sample size; account for intraclass correlation.

Pre-registration template

Copy this and paste into your experiment tracking system or make the fields an interactive form.

Analysis checklist (use before unblinding)

• Confirm randomization integrity and allocation balance across key covariates.
• Verify tracking and metric definitions; sanity-check raw event counts.
• Confirm sample size reached or pre-specified stopping rule met.
• Lock analysis script & version results-producing code.
• Apply planned statistical tests; record effect sizes with confidence intervals.
• Check for unexpected changes in secondary metrics or key segments.
• Document any deviations from pre-registered plan and why.

Guardrails for multiple testing & sequential looks

Multiple comparisons and peeking inflate false-positive risk. Use one or more of these pragmatic options:

• Pre-specify the single primary metric and keep others exploratory.
• Adjust p-values (e.g., Benjamini–Hochberg) when reporting multiple tests.
• Use group-sequential methods or alpha-spending if you plan interim analyses.
• Prefer confidence intervals and effect sizes over noisy p-values; treat small p-values as prompts for replication, not final proof.

Post-experiment interpretation checklist

• Are results statistically and practically meaningful? Report CI and lift magnitude.
• Did secondary metrics or key segments contradict the primary signal?
• Any protocol deviations or instrumentation issues that could explain results?
• Is the finding consistent with prior evidence or theory? If surprising, plan replication.
• Decide and record the action (roll out, iterate, stop, further test), rationale, and next experiment idea.
• Publish results, methods, and data location to your team’s knowledge base for reuse.

Common threats to validity (brief primer)

• Selection bias: non-random assignment or allocation leakage. Mitigation: verify randomization and logging.
• Instrumentation bias: metric or tracking changes during the experiment. Mitigation: lock instrumentation, run audit queries.
• Interference (spillover): treatment effects cross between units. Mitigation: choose appropriate randomization unit or use cluster-randomization.
• Seasonality & external events: confounds from time-based effects. Mitigation: run long enough or use concurrent controls and stratification.
• P-hacking/analysis flexibility: data-driven choices after seeing results. Mitigation: pre-registration and blinded analysis where practical.
• Underpowering: failing to detect meaningful effects. Mitigation: compute realistic MDE and sample size up front; consider pooled or composite metrics if necessary.

Adoption & capability suggestions

This static toolkit is a practical starting point. To make it repeatable and team-friendly, consider:

• Converting the pre-registration template into an interactive form so experiment definitions are captured consistently and stored centrally.
• Adding an interactive sample-size calculator that fills computed sample size into the pre-registration form.
• Implementing the analysis checklist as an interactive checklist that must be completed and saved before unblinding results.
• Publishing experiment outcomes and methods to a shared domain so other teams can inherit and adapt your experiments (versioning and tailoring are valuable).

These enhancements can be implemented using the platform’s Interactive Form Rendering and Data Submission capabilities so responses, decisions, and templates become searchable organizational memory.

Next steps

Copy the pre-registration template into your experiment tracker or ask your librarian to provision an interactive version. Start by pre-registering one experiment and following the analysis checklist to build trusted habits. After 2–3 documented experiments, convert common elements into reusable templates for your organization.