Experiment Runbook: From Hypothesis to Learning

Welcome — Why this runbook exists

This runbook standardizes how teams move from a question to usable learning. It is written for teams running pilots, A/B tests, process experiments, or small-scale trials. Use it to reduce bias, increase reproducibility, and turn short experiments into durable organizational knowledge.

How to use this Playbook

Follow the sections in order when planning an experiment. Keep the pre-registration and analysis plan unchanged while the experiment runs. After completion, use the interpretation and archiving templates to share evidence and make accountable decisions.

Quick overview (the lifecycle)

1. Hypothesis formulation (clear, testable, scoped)
2. Metric selection (primary, secondary, safety)
3. Guardrails & ethics (consent, stopping rules)
4. Experiment specification & pre-registration
5. Data collection & quality checks
6. Analysis (pre-specified)
7. Interpretation & decision commit
8. Evidence archiving and dissemination
9. Scale, replicate, or retire

Hypothesis formulation

Write hypotheses in a structured, falsifiable form. A simple template helps keep everyone aligned.

• State the causal mechanism you expect (why this should work).
• Define the population and context explicitly (who, where, when).
• Set an expected direction and a practical threshold for decision-making.

Metric selection

Choose a single primary metric that directly tests the hypothesis. Add well-justified secondary and safety metrics.

• Primary metric: one clearly defined outcome used for the main decision.
• Secondary metrics: supportive signals, UX measures, leading indicators.
• Safety metrics: customer complaints, error rates, safety incidents, ethical flags.

Record metric definitions precisely (numerator, denominator, filters, time windows). Prefer absolute and relative measures (e.g., percentage point change and percent change).

Guardrails and ethics

• Document consent requirements, privacy impacts, and data retention rules.
• Set exposure limits (max % of users, sites, or customers exposed).
• Define stopping rules for safety or severe negative outcomes.
• Spell out who can pause/stop the experiment and how to escalate.

Experiment specification

Be explicit about design choices so the experiment is reproducible.

• Design: randomized, quasi-experimental, crossover, stepped-wedge, etc.
• Unit of assignment: user, session, location, cohort.
• Control & treatment definitions and implementation details.
• Randomization method and allocation ratio.
• Start/end dates, minimum runtime, and sample size or stopping criteria.
• Infrastructure notes: feature flags, instrumentation owners, rollback plan.

Pre-registration template (use without change while experiment runs)

Data collection checklist

• Confirm instrumentation for each metric (event names, fields).
• Define data capture frequency and retention.
• Implement validity checks: sanity counts, expected ranges, and duplicates.
• Log implementation events (feature flag changes, rollout times).
• Assign a data steward responsible for quality during the run.

Analysis template (pre-specified)

Create an analysis notebook or script that can be re-run to reproduce results.

1. Confirm dataset and time-window used.
2. Data cleaning rules (outliers, missing data handling).
3. Main comparison and statistical test (e.g., difference-in-means, regression, nonparametric).
4. Confidence intervals and effect size reporting; avoid overreliance on p-values.
5. Predefined subgroup or heterogeneity analyses and multiplicity adjustments.
6. Robustness checks (alternative windows, covariate adjustment, falsification tests).
7. Record the exact code and environment used to run analyses.

Interpretation guidance

• Match conclusions strictly to pre-registered hypotheses and metrics.
• Distinguish statistical significance, practical significance, and uncertainty.
• Consider external validity: where and when will this generalize?
• If results are inconclusive, prefer replication or extended samples before scaling.

Decision commit rules

Before running, agree on decision logic so results translate to action without ambiguity.

• Go: primary metric improves by >= X and no guardrail triggered.
• Hold & replicate: effect size smaller than threshold or borderline significance.
• Stop & rollback: guardrail triggered or primary metric declines by >= Y.
• Scale plan: who executes, timeline, monitoring after rollout.

Evidence archiving recipe

Archive everything needed for reuse and auditability.

• Pre-registration document and timestamp.
• Raw and cleaned datasets (with data dictionary).
• Analysis scripts/notebooks and environment specification.
• Implementation logs and deployment records.
• Interpretation memo and decision outcome.
• Tags and metadata: teams, product area, related experiments, lessons learned.

Troubleshooting: common analysis errors

• P-hacking / data dredging: avoid exploring multiple tests and reporting only the one that worked. Keep a record of all tests run and prefer pre-specified analyses.
• Multiple comparisons: adjust for multiple hypotheses or clearly label exploratory analyses.
• Confounding or imbalance: check baseline characteristics and use stratification or covariate adjustment if randomization failed.
• Regression to the mean: beware short pre-post windows; use randomized control when possible.
• Data leakage: ensure features used for analysis were available at decision time and not contaminated by future information.
• Small or unrepresentative samples: treat results as uncertain; replicate in broader contexts before generalizing.
• Overfitting models: prefer simple, interpretable models for primary decisions and reserve complex models for exploration with separate validation.

Reusable artifacts (copy-and-adapt)

Make templates part of your domain library so teams can copy and adapt them. Keep an index of past experiments, decisions, and outcomes to accelerate future learning.

Final checklist before launch

1. Pre-registration completed and timestamped.
2. Instrumentation validated end-to-end.
3. Analysis notebook created and peer-reviewed.
4. Guardrails & stop rules agreed and owners assigned.
5. Archive location provisioned and a steward assigned.

Example (short)

Hypothesis: Shortening checkout increases conversion by >=2pp among new users over 14 days. Primary metric: conversion rate (orders/new users). Design: randomized A/B, 50/50, unit = user, sample = 10k users or 14 days. Analysis: difference-in-means, 95% CI, pre-specified subgroup by device type. Decision: if point estimate >=2pp and CI excludes harm and no guardrail violation, proceed to scaled rollout. Archive pre-registration, raw events, and analysis notebook.

Use this Playbook as a living document. When a useful pattern or checklist emerges, add it to your organization's Experimentation Toolkit so future teams learn faster and avoid repeating mistakes.