Checklist

Data Product Readiness Checklist for Learning Use Cases

A practical checklist to evaluate whether a dashboard, dataset, or analysis is dependable and actionable for experiments, huddles, and measurement-driven decisions. Includes observable acceptance criteria, examples of evidence to collect, and recommended validation experiments.

Purpose

This checklist helps teams decide whether a data product (dashboard, dataset, metric, or analysis) is ready to be used for learning: running experiments, guiding huddles, or making operational decisions. Use it to diagnose gaps that reduce trust, block experiments, or slow learning cycles.

How to use this checklist

For each item below, record one of: Yes (meets criteria), Partial (some criteria met), or No (does not meet). For any Partial/No, add brief Evidence and Next Steps. Where useful, attach links to documentation, queries, or tickets.

Scoring guidance

  • Yes — the item has clear, documented acceptance criteria and living evidence (examples, queries, tests, links).
  • Partial — some documentation or tests exist, but gaps remain that could affect decisions.
  • No — absent or misleading; not safe for decision-making or experiments.

Checklist

1. Metric definitions & semantics

Goal: A human and machine-readable definition exists so different teams interpret the metric the same way.

  • Defined metric name and clear plain-language description (what it measures and why).
  • Formal definition: exact formula or SQL, including aggregation window, filters, join logic, and dimensionality.
  • Known edge cases documented (e.g., how to handle nulls, duplicates, partial sessions).
  • Example queries and sample rows for quick validation.

Evidence: Link to metric spec, canonical SQL, sample query / notebook.

2. Source lineage & transformations

Goal: You can trace the metric from user-visible value back to raw events and transformation steps.

  • List of upstream tables/events and columns used.
  • ETL/job names and versions that produce the dataset or derived table.
  • Documented transformation logic and any business rules applied.
  • Known assumptions or sampling applied to source data.

Evidence: Data catalog entry, transformation DAG link, or README.

3. Data quality tests, acceptance criteria, & severity

Goal: Automated checks exist and failures are triaged according to business impact.

  • Tests cover freshness, completeness, null rates, ranges, uniqueness, and expected distributions.
  • Acceptance thresholds and severity levels (warning vs. blocking) are defined.
  • Tests run automatically (CI, scheduled jobs) and results are visible to stakeholders.
  • Recent test history is available (last N runs) and shows stability or known incidents.

Evidence: Links to test dashboards, alerts, or pipeline job logs.

4. Ownership & operability

Goal: Clear human and team ownership, with an agreed escalation path and runbook.

  • Designated data product owner (person/team) and contact details.
  • Data steward responsible for upstream instrumentation and tagging.
  • Runbook: how to respond to data quality failures, who to notify, and rollback/backfill procedures.

Evidence: Owner listed on catalog entry and an open runbook or incident playbook.

5. Refresh cadence, latency & SLAs

Goal: Refresh cadence matches the needs of intended decisions and experiments.

  • Documented refresh frequency (real-time, hourly, daily, weekly) and last successful run time.
  • Latency expectations and acceptable staleness for the learning use case.
  • Backfill policy and how late-arriving data is handled in metrics and experiments.

Evidence: Pipeline schedule, SLA statements, and example timestamps showing freshness.

6. Access controls & privacy

Goal: Only authorized users can access sensitive data; PII is handled correctly for learning use cases.

  • Access level required (who can view, who can query) and how permissions are managed.
  • PII/PHI handling: masking, pseudonymization, or aggregation policies applied and documented.
  • Data retention and compliance requirements declared.

Evidence: Access policy link, example role permissions, and data protection notes.

7. Instrumentation coverage & gaps

Goal: Event taxonomy and instrumentation provide the signal necessary to support the metric and experiments.

  • Map of required events/attributes and whether they are instrumented end-to-end.
  • Sampling rates and whether sampling affects metric accuracy.
  • Known instrumentation gaps or race conditions that could bias results.
  • Instrumentation tests (unit or integration) are present and pass.

Evidence: Instrumentation matrix, QA test results, or instrumentation tickets.

8. Labeling for learning use

Goal: The data product includes a short statement of intended decisions, confidence level, and contexts where it applies.

  • Primary decisions the metric should inform (e.g., launch go/no-go, product prioritization, ops alerts).
  • Contexts where metric is valid (user segments, countries, device types) and where it's not.
  • Estimated uncertainty or confidence band, and major known biases.

Evidence: Label on the dashboard/dataset showing decisions supported and confidence notes.

9. Recommended validation experiments

Goal: Practical ways to validate the metric and confirm it supports correct learning and decisions.

  • Backtest: compare metric over historical experiments or known outcomes to check signal alignment.
  • Shadow/parallel run: run new metric computation in parallel with current production metric to compare results before switching.
  • Sanity A/B checks: small experiment to see if metric moves as expected when controlled changes are applied.
  • Manual audit: sample-level reconciliation against source systems or customer support logs for critical segments.

Evidence: Links to experiment plans, backtest notebooks, or reconciliation reports.

10. Discoverability & documentation

Goal: Teams can find the data product, understand it quickly, and know how to use it.

  • Data catalog entry with tags (metric name, owner, decision area, freshness).
  • Concise README: purpose, definition, typical uses, limitations, and links to related artifacts.
  • Changelog or version history for major definition changes.

Evidence: Catalog entry link and README location.

11. Monitoring, alerting & change management

Goal: Teams are alerted to regressions and schema or logic changes go through a controlled process.

  • Metric drift and test failure alerts are configured and routed to the owner/channel.
  • Change process for metric definition: review, testing, and stakeholder sign-off before release.
  • Versioned artifacts so previous definitions can be restored or compared.

Evidence: Alerting rules and change approval workflow links.

12. Known issues & mitigations

Goal: Transparent record of remaining risks and interim mitigations so decision-makers weigh them appropriately.

  • List of known limitations, expected impact, and planned remediation steps.
  • Temporary workarounds or guardrails for experiments or huddles while issues are resolved.

Evidence: Issues list, tickets, and planned timeline for fixes.

Deliverables & next steps

  1. If any item is Partial/No: create an explicit next-action ticket with owner and target date.
  2. Run at least one recommended validation experiment (shadow run or backtest) before trusting a metric for high-stakes decisions.
  3. Publish or update the data catalog entry with owners, links, and an explicit "decisions supported" label.
  4. Schedule a short huddle with the owner, data steward, and one downstream decision-maker to review the checklist and agree on go/no-go for experiments.

Quick checklist table (copyable)

Use this compact checklist to capture Yes / Partial / No and evidence links when assessing a data product.

  • Metric definitions & semantics: _____ Evidence: _____
  • Source lineage & transformations: _____ Evidence: _____
  • Data quality tests & severity: _____ Evidence: _____
  • Ownership & runbook: _____ Evidence: _____
  • Refresh cadence & SLA: _____ Evidence: _____
  • Access controls & privacy: _____ Evidence: _____
  • Instrumentation coverage: _____ Evidence: _____
  • Labeling for learning use: _____ Evidence: _____
  • Recommended validation experiments: _____ Evidence: _____
  • Discoverability & documentation: _____ Evidence: _____
  • Monitoring & change management: _____ Evidence: _____
  • Known issues & mitigations: _____ Evidence: _____

Notes: This checklist preserves the original intent—ensuring data assets are trustworthy and actionable for experiments and decisions—while adding concrete acceptance criteria, evidence examples, and pragmatic validation experiments. Tailor fields and evidence types to your organization's tools (data catalog IDs, pipeline names, alert channels).

Discuss Ask the Librarian

Discussion

Comments and conversation will live here.