Data Readiness & Leakage Checklist

This checklist helps you decide whether your data supports a reliable predictive pilot and highlights the common problem of data leakage—when the model has access to information it would not realistically have at decision time.

Basic availability

• There is a clearly defined outcome (label) recorded reliably in historical data.
• Every labeled example includes a timestamp for the outcome event.
• Entities have stable unique identifiers used across records (customer_id, machine_id).

Temporal alignment (avoid leakage)

• All features used for training are available before the decision timestamp in held-out examples.
• Check for fields that are generated after the outcome (e.g., 'time_to_resolution', 'closed_by') and exclude them.
• Prefer event-based windows (lookback windows) and time-based train/validation splits for anything with time ordering.

Label quality

• Labels represent the actionable event you care about, not a proxy that was created by later manual work.
• Label definitions are stable over the historical period used—if not, annotate or restrict the timeframe.

Sampling and bias

• Check for sampling that would bias historical outcomes (e.g., only recorded failures when equipment was monitored more closely).
• Confirm the historical baseline and any manual interventions that could have affected the recorded outcomes.

Missingness and signal quality

• Measure missingness by field and by time. Investigate patterns that align to operations or instrumentation changes.
• Verify sensors, logs, or sources have stable schemas and clear error codes.

Feature reproducibility

• You (or your pipelines) can reproduce feature calculations at inference time with the same logic and timestamps.
• Store deterministic feature code, feature creation dates, and dataset versions for auditing.

Governance and privacy

• Confirm consent, regulatory, and privacy requirements are identified and documented.
• Restrict access and consider pseudonymization for sensitive fields during model development.

Quick tests to run

• Temporal holdout test: train on older period, test on newer period and confirm performance is consistent.
• Leakage probe: add a random lookahead feature (e.g., future_value) to training and ensure it improves nothing.
• Label drift check: compare outcome rates across training and current period.

Next steps if the checklist exposes gaps: narrow the pilot scope to a period or subgroup where data is clean, instrument missing data proactively, or design an experimentation strategy (collect labels in parallel) before wide model rollout.