Template

Experiment Registry Template (Canonical Entry)

A canonical, decision-linked template for recording experiments, results, artifacts, and actionable learnings so teams can search, trust, and reuse evidence. Includes field definitions, metadata and tagging guidance, an evidence-grading rubric, analysis and governance checklists, and an example entry.

Purpose

This canonical experiment registry template captures the essential information teams need to make experimental learning discoverable, trustworthy, and actionable. Use it to record hypotheses, design, measurements, analyses, artifacts, decisions, and follow-ups in a consistent format that preserves context and supports reuse.

When to use

Use this template for any planned test, pilot, trial, A/B experiment, operational change, or observational evaluation where the outcome will inform decisions. It is suitable for product teams, operations, process improvement, research projects, safety trials, and service experiments.

How to use

  1. Record key metadata and ownership before the experiment begins.
  2. Pre-specify the hypothesis, primary metric, guardrails, and analysis plan where practical.
  3. Attach artifacts (scripts, datasets, runbooks) and link related decisions so context is preserved.
  4. Grade evidence quality after analysis and record recommended next actions with owners and due dates.

Canonical Fields (required and recommended)

Core fields

  • Experiment name — concise, unique title (include project prefix if used by the organization).
  • Experiment ID — stable identifier for cross-referencing (e.g., EXP-2026-045).
  • Owner / Responsible party — person(s) accountable for the experiment record and follow-through (name, role, team).
  • Status — draft / running / completed / abandoned / archived.
  • Start date and end date (actual or planned).

Hypothesis & purpose

  • Hypothesis — a short, testable statement describing expected change and why (What change? For whom? By how much?).
  • Goal / Business outcome — what decision or business problem this experiment is intended to inform.

Metrics & guardrails

  • Primary metric — single headline metric that determines success (define numerator/denominator and time window).
  • Secondary metrics — other important outcomes to monitor (list and define).
  • Guardrails / Safety constraints — metrics or conditions that trigger immediate rollback or stop the experiment (e.g., error rate > X, safety incidents, revenue drop thresholds).
  • Minimum detectable effect & sample size — planned MDE and sample size calculation or rationale (or qualitative stopping rules for non-statistical tests).

Design & execution

  • Experiment type — A/B, randomized controlled, cohort comparison, pilot, observational, etc.
  • Design summary — grouping, randomization method, control conditions, timelines, and assignment rules.
  • Pre-registered analysis plan — high-level plan including primary comparisons, statistical tests, metrics, and pre-specified subgroup analyses. Note any planned interim looks and stopping rules.
  • Sample sources — populations, segments, geographic or temporal constraints, inclusion/exclusion criteria.

Data & analysis

  • Data sources — databases, pipelines, tracking tags, and owners responsible for each source.
  • Analysis notes — raw results, calculations, key tables, p-values, confidence intervals, effect sizes, practical significance, and any data cleaning or exclusion rules.
  • Sensitivity checks — results of robustness checks and alternative specifications.
  • Limitations — threats to validity, confounders, or missing context that readers should know.

Outcome & evidence

  • Outcome summary — succinct conclusion tied to the primary metric (what happened vs. hypothesis).
  • Evidence grade — assign High / Moderate / Low (see rubric below) and justify.
  • Decision link — reference to the decision this experiment supports (Decision ID, summary, decision date, and final action taken).

Artifacts

  • Artifacts (links) — code, dashboards, raw data extracts, runbooks, analysis notebooks, pre-registration docs, consent forms. For each artifact include: link, type, short description, owner, and access level.

Follow-up & governance

  • Lessons learned — what worked, what didn't, unexpected observations, and practical implications.
  • Recommended next actions — concrete actions (e.g., roll out, scale, re-run, stop), owner, due date, priority, and status.
  • Retention & privacy — retention period for artifacts and data classification (public/internal/confidential/sensitive) and any required data deletion actions.
  • Version history — record of edits, who changed what and when; include links to prior versions if archived.

Search & discovery metadata

  • Tags / taxonomy — product/team, experiment type, hypothesis area, technology stack, skillset, geographic site, customer segment. Use controlled vocabulary where possible.
  • Keywords — free-text keywords to improve discovery for cross-team searches.
  • Related experiments — links to experiments that informed or depend on this work.

Evidence Grading Rubric (recommended)

Use an explicit rubric so readers can quickly judge reliability.

  • High — pre-registered, randomized or otherwise well-controlled design, sufficient sample size, statistically and practically significant results, artifacts and raw data available for review, and reproducible analysis steps.
  • Moderate — clear design and analysis, plausible evidence but with some limitations (smaller sample, quasi-experimental design, or partial artifacts). Useful for directional decisions with caution.
  • Low — anecdotal, small-n, poorly controlled, or missing core artifacts; useful for hypothesis generation but not for firm decisions without further testing.

Analysis Checklist

  • Was the hypothesis and primary metric stated before viewing results?
  • Are data sources, cleaning rules, and exclusions documented?
  • Were pre-specified statistical methods used? If not, document deviations and why.
  • Were sensitivity analyses performed and reported?
  • Are raw artifacts attached or linked and accessible to reviewers?
  • Is an evidence grade assigned and justified?
  • Is a clear next action recorded with owner and due date?

Governance & Lifecycle

  • Ownership — every experiment must have a named owner responsible for accuracy and follow-through.
  • Pre-registration — encourage pre-registering key items before running tests to reduce bias.
  • Review process — define who reviews completed experiments (peer review, data steward, governance committee) and what criteria are required to accept evidence into the canonical repository.
  • Retention & archiving — set default retention periods; document rules for archiving or deleting sensitive data.
  • Governance flags — allow reviewers to flag experiments as "reproducible", "needs follow-up", or "do not reuse" with reasons.

Tagging & Searchability Guidance

Adopt a small controlled vocabulary for common dimensions (team, product, experiment-type, outcome-area). Examples:

  • team:payments, team:fulfillment
  • type:ab-test, type:pilot, type:observational
  • area:conversion, area:quality, area:safety

Also include free-text keywords for edge cases. Ensure each entry includes at least 3 searchable tags (team/product, type, area).

Example entry (short)

Experiment name: EXP-2026-045 - Reduce checkout friction for returning customers

Owner: Alex Rivera, Growth Team

Hypothesis: Pre-populating saved addresses for returning customers will increase checkout completion rate by at least 5% within 30 days.

Primary metric: 30-day checkout completion rate (completed checkouts / checkout sessions).

Design: A/B test randomized by user id, 50/50 split, run for 4 weeks or until minimum sample size (N=10,000 per variant) reached. Interim checks permitted only for safety guardrails.

Guardrails: Increase in payment failure rate > 1% triggers immediate rollback.

Outcome summary: Completion rate +6.1% (p=0.02). Payment failures unchanged. Evidence grade: High.

Recommended next actions: Roll out to 100% of returning customers in phased deployment. Owner: Product Ops. Due: 2026-07-15.

Implementation notes & suggested interactive features

As a template this item is designed for HTML rendering. To improve usability and institutional memory, consider converting the template into an interactive form that:

  • enforces required fields (Experiment ID, Owner, Hypothesis, Primary metric, Evidence grade, Recommended next action),
  • validates date and numeric fields (sample size, p-values),
  • collects artifacts with structured metadata (type, link, owner), and
  • stores submissions as JSON to enable dashboards, cross-experiment queries, and automated alerts for guardrail breaches.

See CapabilityEnhancementNotes for platform-specific suggestions (interactive rendering and JSON storage).

Quick checklist before publishing

  • Owner, hypothesis, primary metric and pre-registered plan are recorded.
  • Artifacts and data access instructions are attached.
  • Evidence grade assigned with justification.
  • Decision link and recommended next action with owner and due date are present.
  • Tags and metadata applied for discoverability.
Discuss Ask the Librarian

Discussion

Comments and conversation will live here.