Experiments Repository Schema & Evidence Bank

Experiments Repository Schema & Evidence Bank — Canonical Reference

Purpose: Provide a clear, reusable schema and practical guidance so teams can record experiments, preserve context and artifacts, grade evidence quality, and link outcomes to decisions. The goal is reliable discoverability, repeatability, and reuse across projects and teams.

Core schema (fields and recommended types)

1. experiment_id (string): Stable unique identifier, e.g., "exp-2026-0034". Use a team/namespace prefix where useful.
2. title (string): Short descriptive title focused on the tested hypothesis.
3. hypothesis (text): Clear, testable hypothesis. Prefer the format "If <action> then <measurable outcome> because <rationale>."
4. owner (object): {name, role, team, contact}. Identify the accountable owner for follow-up and replication.
5. start_date, end_date (date): ISO 8601 dates. If ongoing, leave end_date null and set status accordingly.
6. status (enum): planned / running / completed / abandoned / replicated
7. primary_metric (object): {name, direction (increase/decrease), unit, target_delta}. The single most important metric for judging the hypothesis.
8. secondary_metrics (array of objects): Additional metrics to observe for tradeoffs or signals.
9. power_sample_notes (text): Short note on sample size, statistical power, minimum detectable effect, randomization method, and segments used.
10. segments (array): Which cohorts, regions, or device types were included or excluded.
11. design_summary (text): Brief method description (A/B, multi-arm, before/after, lab study, qualitative diary study) and key controls.
12. experiment_artifacts (array of links/objects): notebooks, dashboards, datasets, runbooks, scripts, raw files. Include storage location, version, and access instructions.
13. outcome_summary (text): Short plain-language outcome: what happened and how it compared to expectations.
14. analysis_notes (text): Key statistical tests, p-values, uncertainty, caveats, and analyst name.
15. evidence_grade (enum + metadata): high / medium / low — see rubric below. Include reason and evidence_date.
16. decision (object): {verdict: keep/modify/kill/hold, decision_date, decision_by, decision_context, next_steps_link}
17. replication_notes (text): Steps needed to replicate, known obstacles, environment differences, and whether a replication has been attempted or succeeded.
18. related_projects (array of ids): Links to related tickets, OKRs, initiatives, or experiments.
19. tags (array): Short keywords for discovery (customer-type, technology, hypothesis-pattern).
20. retention_policy (string): e.g., "archive-after-2y" with rationale and privacy level.
21. privacy_level (enum): public/internal/confidential — informs search and sharing rules.

Example JSON instance

{
  "experiment_id": "exp-2026-0034",
  "title": "Homepage CTA color: increase signups",
  "hypothesis": "If we change CTA to high-contrast orange, then signup rate will increase by >=5% because the color improves visibility.",
  "owner": {"name": "A. Rivera", "role": "Product Manager", "team": "Growth"},
  "start_date": "2026-02-01",
  "end_date": "2026-02-15",
  "status": "completed",
  "primary_metric": {"name": "signup_rate", "direction": "increase", "unit": "%", "target_delta": 5},
  "secondary_metrics": [{"name":"CTR","direction":"increase","unit":"%"}],
  "power_sample_notes": "Target N=50k visitors; MDE=4.8%; randomization by user cookie.",
  "segments": ["new_visitors","mobile"],
  "design_summary": "A/B test, 50/50 split, canonical traffic, 2-week run.",
  "experiment_artifacts": [{"type":"notebook","link":"/artifacts/exp-2026-0034-analysis.ipynb","version":"v1"}],
  "outcome_summary": "Signup rate increased 6.1% (p=0.03).",
  "analysis_notes": "Two-sided t-test, controlled for time-of-day. Excluded bots.",
  "evidence_grade": {"level":"high","reason":"stat sig + adequate power"},
  "decision": {"verdict":"keep","decision_date":"2026-02-18","decision_by":"Growth leadership","next_steps_link":"/playbooks/cta-deploy"},
  "replication_notes": "Deploy to EU on mirrored test for 2 weeks; monitor for localization effects.",
  "tags": ["cta","homepage","growth"],
  "retention_policy": "archive-after-2y",
  "privacy_level": "internal"
}

Minimal UI-friendly card layout (for quick browsing)

Present experiment cards in search results with the following elements so readers can scan and prioritize:

Import / Export guidance

CSV header suggestions for bulk import/export: experiment_id,title,hypothesis,owner_name,owner_team,start_date,end_date,status,primary_metric_name,primary_metric_target,primary_metric_unit,outcome_summary,evidence_grade,decision_verdict,decision_date,tags,privacy_level,artifact_links

Evidence grading rubric (practical)

• High — Adequate sample size/power, statistically significant effect or strong qualitative consistency; artifacts and raw data available; analysis reproducible.
• Medium — Directionally consistent result but limited power, incomplete artifacts, or potential confounders noted.
• Low — Underpowered, informal observation, single-subject, or missing reproducible artifacts; useful as a hypothesis but not decision-grade evidence.

Decision-link practices

Every experiment record that leads to a decision should include:

1. Explicit decision object (verdict, date, decision_by).
2. Link from decision to deployment playbook, change ticket, or follow-up experiment.
3. Rationale documenting how the experiment informed the decision, including known limitations.

Governance, retention, and access

• Assign an owner for each record responsible for replication and follow-up.
• Define retention policy by privacy_level and business value; archive stale records but keep searchable metadata.
• Enforce minimal required fields on submission (experiment_id, title, hypothesis, owner, status, primary_metric, outcome_summary or plan) to avoid orphaned skeletons.

Submission checklist (use when saving a new record)

• Is the hypothesis testable and explicit?
• Is the primary metric defined and measurable?
• Are artifacts (analysis notebook, raw data pointer) attached or linked?
• Is owner and expected replication date assigned?
• Is privacy level set and retention policy noted?

Search and discovery (recommended facets)

Provide filters for status, owner/team, tags, primary_metric, evidence_grade, privacy_level, date range, and related projects. Allow free-text search across hypothesis, outcome_summary, and analysis_notes.

Replication & reuse tips

• Keep artifacts versioned and point to the exact dataset snapshot used in analysis.
• Record environment or configuration variables that materially affect results.
• When reusing a result in a different context, add a replication record documenting contextual differences and new evidence_grade.

Where this fits in a living domain

This canonical schema is intended as a baseline for site- or enterprise-specific adaptations. Teams should copy and tailor required fields, privacy rules, evidence rubrics, and UI card designs to their operating context while preserving key interoperability fields (experiment_id, owner, decision, artifacts) so records remain meaningfully discoverable and shareable.