Operational Intelligence Runbook: From Anomaly to Investigation

Purpose

This runbook links monitoring signals to a reliable investigation and resolution workflow so alerts lead to fast, clear action and fewer repeat incidents. Use it to triage, investigate, record findings, assign countermeasures, and verify fixes with post-implementation monitoring.

Scope & Audience

Applies to operational dashboards, automated alerts, and anomaly detections across monitoring, telemetry, logs, and business metrics. Intended for on-call responders, SREs, ops analysts, data owners, quality leads, and shift supervisors.

Key Roles

• Alert Owner: The team or individual responsible for the system that generated the alert.
• Investigator: Person assigned to run triage and coordinate the investigation.
• Action Owner: Person responsible for implementing countermeasures and tracking fixes.
• Incident Coordinator (when escalated): Manages cross-team communication and follow-up.

Severity and Investigation Depth (Decision Criteria)

Classify the alert to determine how deep and fast the response must be. Adapt these definitions to your environment.

• Sev-Critical — Immediate business/customer impact, data loss, safety, or regulatory exposure. Escalate and mobilize within minutes.
• Sev-High — Noticeable degradation or repeated failures affecting many users or core processes. Triage promptly and remediate quickly.
• Sev-Medium — Localized or non-blocking issues that require investigation but not immediate escalation.
• Sev-Low — Informational anomalies, cosmetic issues, or suspected false positives; schedule normal work for resolution.

Investigation depth increases if the alert meets any of: customer impact, safety/regulatory risk, repeated occurrences, cross-system correlation, or unknown root cause after basic checks.

Quick Triage Checklist (first 15–60 minutes)

1. Validate the alert (data sanity)
   • Confirm the alert source and timestamp. Check for known deploys, schema changes, maintenance windows, or data-source outages.
   • Run a quick query to confirm the metric or log pattern. Is the anomaly reflected in raw data?
   • Check related signals and dependencies for correlation (upstream/downstream systems).
2. Capture evidence
   • Save screenshots, metric query snapshots, raw log extracts, query text, time range, and any suspicious node/host IDs.
   • Note environment (prod, staging), affected customers/regions, and the alert rule conditions.
3. Notify owners
   • Use the defined escalation list to inform the Alert Owner and primary on-call. Include severity, one-line impact, and link to evidence.
4. Open investigation ticket using the ticket template below. Link evidence and include initial triage notes.
5. Decide investigation depth — follow the severity decision rules above. For Sev-Critical, assemble incident coordinator within predefined SLA.

Evidence Checklist (minimum)

• Alert ID and rule definition (link)
• Time window and timezone for observed anomaly
• Raw query or log snippet showing anomaly
• System/node identifiers and version/deploy info if applicable
• Screenshot of dashboard and relevant graphs
• Any recent config or release notes that could explain change

Investigation Steps (detailed)

1. Reproduce & scope

   Can you reproduce the anomaly by re-running queries or replaying logs? Determine affected time range, customers, services, and geographic scope.

2. Isolate root cause candidates

   Use rapid techniques: compare timestamps across services, check recent deploys, evaluate upstream input quality, validate schema/ETL jobs, and check sensor/agent health.

3. Decide temporary mitigations

   If immediate harm is possible, implement short-term countermeasures (rate limiting, pausing a job, failover to cached data, revert recent deploy) and record rollback steps.

4. Root cause analysis

   When initial isolation suggests the cause, capture evidence and use structured RCA techniques (5 Whys, fault tree, or fishbone). Record assumptions and how they were validated.

5. Assign countermeasures

   Document corrective actions, owners, due dates, and acceptance criteria. Distinguish between temporary mitigations and permanent fixes.

6. Implement fix and verify

   Implement permanent change in controlled manner (PR, change request) and validate with the same metrics that detected the anomaly. Avoid relying only on absence of alerts—use targeted tests and user-impact checks.

7. Post-implementation monitoring

   Schedule monitoring windows and define success metrics (no recurrence for X days, error rate under Y, throughput restored). Close ticket only after acceptance criteria are met.

Ticket Template (fields to populate)

• Title: [Short description] — [AlertID] — [Severity]
• Detection time (UTC)
• Alert source & rule (link)
• Environment (prod/staging)
• Initial impact statement (who/how long)
• Evidence links & attachments
• Investigator & Alert Owner
• Initial triage notes
• Root cause hypothesis
• Temporary mitigations (if any)
• Permanent corrective actions, owners, due dates
• Acceptance criteria & monitoring plan
• Post-mortem summary and lessons learned (populate on close)

Communication Examples

Keep messages concise. Include severity, one-line impact, and a link to the ticket/evidence.

Alert notification (chat/email): "Sev-High: 300ms latency spike on Payments API (region EU) since 09:22 UTC. Customers may see checkout delays. Investigator assigned: @jane. Ticket: #12345 (link)."

Incident update: "Update: root cause appears to be a queued job backlog after deploy v2.4. Temporary mitigation applied (backfill job paused). Working on permanent fix; ETA 12:00 UTC. See ticket #12345 for details."

Example Timeline SLAs (adapt to your org)

• Acknowledge alert: within 15 minutes (Sev-Critical) / 60 minutes (Sev-High) / 4 hours (Sev-Medium).
• Initial triage & evidence capture: within 1 hour (Sev-Critical/High).
• Temporary mitigation decision: within 2 hours (Sev-Critical) / 12 hours (Sev-High).
• Investigation plan with owners and due dates: within 4 hours (Sev-Critical) / 24 hours (Sev-High).
• Permanent fix delivery target: negotiated per issue (example: 7–14 days for code fixes; 24–72 hours for config/ops fixes).
• Post-implementation monitoring window: typically 72 hours to 14 days depending on risk and recurrence potential.

Common False-Positive Checks

• Is the data pipeline healthy? (ingestion lag, ETL failures)
• Were there recent deploys, config changes, or schema updates?
• Is there a timezone or daylight savings mismatch in timestamps?
• Sensor/agent health and connectivity checks
• Sampling or aggregation artifacts in dashboards (look at raw-level data)

Closing the Loop

On close, complete a short post-mortem: summarize root cause, list corrective actions and owners, document what prevented timely detection or response, and identify one improvement to reduce recurrence (alert tuning, runbook update, automated test, or new guardrail).

Quick Reference: What to do in the first 15 minutes

1. Confirm alert authenticity (run raw query).
2. Capture and link evidence.
3. Notify owner & assign investigator.
4. Open ticket with minimal required fields.
5. Decide whether to escalate or apply an immediate mitigation.

Adapt this runbook to local teams, tooling, and SLAs. Keep the ticket template and evidence checklist updated as dashboards, alert rules, and systems evolve.