Checklist

Instrumentation Audit Checklist for Learning Use Cases

An interactive, savable checklist teams can use to audit instrumentation readiness for experiments and dashboards. Each item includes a short 0–3 scoring rubric, a recommended owner field, and a notes field for evidence and next steps.

Interactive Tool

Instrumentation Audit Checklist for Learning Use Cases

This checklist helps teams evaluate whether the data, events, and metadata needed for experiments and measurement-driven learning are captured and trustworthy. For each item use the 0–3 rubric (0 = missing, 1 = partial/incorrect, 2 = implemented but incomplete, 3 = complete and verified). Record the recommended owner for remediation and paste evidence links or notes. Save results so your team can track progress over time.

Scoring rubric summary: 0 Not instrumented or undocumented; 1 Instrumented in a limited or inconsistent way; 2 Instrumented with basic docs, but missing ownership/alerts/lineage; 3 Fully instrumented, schema documented, owner assigned, alerting and retention defined, and lineage visible.

Enter the date you performed this audit.
Person performing the checklist.
e.g., Web App, Mobile iOS, Payments, Recommendation Engine.
0-3 rubric. Consider experiment-relevant events: exposures, impressions, conversions, failures, important UX milestones.
1.0 10.0
Person or team responsible for ensuring events are produced and accurate.
Links to example logs, event payloads, dashboard screenshots, or ticket IDs.
Is there a living schema (field list, types, example payloads) in your data catalog or doc site? 0-3 rubric.
1.0 10.0
Who maintains the schema and approves changes?
Are events timestamped consistently (incl. timezone), and do they include stable user identifiers suitable for aggregation and experiment assignment?
1.0 10.0
Who ensures ID consistency and timestamp correctness?
Are error states and important edge cases instrumented (retries, timeouts, partial responses)? 0-3 rubric.
1.0 10.0
Owner responsible for monitoring and triage.
Are retention policies, PII flags, and privacy classifications attached to events/data tables? 0-3 rubric.
1.0 10.0
Data custodian or privacy officer responsible for compliance and retention rules.
Do teams know if data is sampled, what cardinality limits exist, typical pipeline latency, and how these affect experiments? 0-3 rubric.
1.0 10.0
Person who can confirm sampling rates and SLAs.
Are there alerts for ETL failures, schema changes, drops in event volume, or stale dashboards? 0-3 rubric.
1.0 10.0
Team who receives and acts on pipeline or metric alerts.
Can you trace a metric from dashboard back to raw events and transformations? Are transformations documented/approved? 0-3 rubric.
1.0 10.0
Owner for data lineage documentation (e.g., data engineering, analytics engineering).
Are there clear contracts (SLAs, expected fields, schemas) between producer and consumer teams? 0-3 rubric.
1.0 10.0
Who maintains the contract and mediates schema changes?
Are dashboards and experiment analyses using the same metric definitions and filters? Are edge-case filters documented? 0-3 rubric.
1.0 10.0
Owner of the canonical dashboard or analytics definition.
Optional. Compute an average of the numeric scores above or use your own weighted approach.
List concrete next steps with owners and target dates (e.g., 'Add timestamp to X event — owner Y — due 2024-09-30').
Paste links to dashboards, data catalog entries, runbooks, or ticket numbers that support your scoring.
Select 'Yes' if remediation work or data engineering changes are required.
You can explore this tool now. Sign in or create an account to save your responses and return to them later.
Sign In to Save Create an Account
Make this tool part of your work

Save a personal copy, bring it to your team, or tailor the questions and workflow to fit what you are hungry to improve.

Member customization and team collaboration are coming soon.
Discuss Ask the Librarian

Discussion

Comments and conversation will live here.