Data Readiness & Bias Checklist

Before you rely on a dataset to guide a decision or an experiment, run this quick checklist. It’s meant to be practical and fast—use the items as pre-meeting checks or a short team review.

1. Is the metric well-defined?

   Confirm the exact definition (numerator, denominator, time window). Ambiguous metrics lead to arguments, not decisions.

2. Can we reproduce the metric?

   Someone should be able to re-run the query or process and get the same number.

3. Is there a clear baseline?

   Do we have recent historical values and understanding of seasonality or recent changes that affect the baseline?

4. Are the data sources reliable?

   Note any known gaps, sampling issues, or upstream processes that may introduce bias.

5. Could measurement change with the intervention?

   Consider instrumentation or reporting changes that could create apparent effects unrelated to the intervention.

6. Have we considered confounders?

   List major factors that could influence the metric besides your action (e.g., marketing campaigns, seasonality, policy changes).

7. Is the sample size sufficient for the expected effect?

   If running an experiment, estimate whether you can observe the expected change in the available time or population.

8. Do we have a monitoring plan?

   Identify dashboards, update cadence, and escalation steps for negative signals.

9. Have we documented assumptions?

   Record assumptions and potential biases so future reviewers know what you expected and why.

Use this checklist to reduce surprises and to make the evidence you collect more trustworthy and more useful for decisions.