Operational Intelligence Reference Architecture

Operational Intelligence Reference Architecture

This reference provides a repeatable, implementation-focused architecture for turning operational (edge/OT) data into trusted dashboards, alerts, and decisions. It explains the layers, who owns them, key design decisions, and pragmatic guidance on latency, retention and lineage so teams can avoid brittle, ad-hoc solutions.

Architecture overview

The architecture is organized into composable layers. Each layer has clear responsibilities, quality constraints, and handoff points so teams can independently own and evolve components without losing trust in the data.

Layers and responsibilities

• Data capture (sensors, PLCs, controllers)
  • Primary responsibility: accurate, timely measurement and event emission at the source.
  • Design notes: prefer standardized protocols (OPC-UA, Modbus/TCP, MQTT/TS, EtherCAT where appropriate) and stable asset identifiers. Tag units and semantic meaning at the source when possible.
• Edge preprocessing
  • Primary responsibility: protocol bridging, buffering, local aggregation, first-pass validation, and deterministic low-latency decisions (safety or shutdown logic remains local).
  • Design notes: keep transformations minimal and reversible. Record metadata about dropped or filtered samples. Implement local health telemetry to detect sensor or network problems early.
• Secure ingestion & transport
  • Primary responsibility: reliable, authenticated, and auditable transport from edge to central systems (message brokers, gateways).
  • Design notes: use mutually authenticated TLS, message sequencing or idempotent writes, and durable queuing to tolerate network loss. Choose protocols that meet latency and scale needs (MQTT/Kafka/HTTP).
• Time-series storage & archival
  • Primary responsibility: efficient storage of high-cardinality, timestamped measurements and events with clear retention and aggregation strategies.
  • Design notes: use a purpose-built TSDB for high ingest/queries (InfluxDB, Timescale, Prometheus for metrics) and a data lake or object store for raw, long-term archives. Store raw ingested data and later-produced aggregates separately.
• Analytics & ML
  • Primary responsibility: generate insights, features, predictions and derived KPIs. Support both batch and real-time pipelines.
  • Design notes: separate feature engineering (repeatable pipelines) from model serving. Apply model governance (versioning, A/B testing, performance monitoring) and ensure models can be traced back to input data and preprocessing steps.
• Alerting & orchestration
  • Primary responsibility: convert signals into prioritized, actionable notifications with operator context and remediation steps.
  • Design notes: implement deduplication, suppression windows, alert confidence scoring, owner assignment, and clear runbooks. Always attach context (asset, recent trends, likely causes) and show the data lineage that led to the alert.
• Dashboarding & decision UX
  • Primary responsibility: present actionable views that lead to decisions, not just data.
  • Design notes: design dashboards around roles and actions (operator, supervisor, reliability engineer). Include clear SLAs, owners, and links to the underlying evidence and runbooks. Provide concise KPI tiles plus drilldowns for root-cause exploration.
• Governance, catalog & lineage
  • Primary responsibility: data cataloging, access control, retention policies, transformation lineage, and audit trails.
  • Design notes: maintain a schema/semantic registry for assets and signals, record every transformation step, and make lineage discoverable from dashboard elements and alerts.

Key cross-cutting concerns

Latency

Define latency requirements by use case: local control (<100 ms) stays on the edge; operational dashboards and alerts often need 1–10s; analytics and trend detection can tolerate minutes. Design each pipeline to meet the strictest downstream requirement and document expected end-to-end latency.

Retention

Adopt a tiered retention policy: raw high-resolution data (hot) for short windows (7–30 days), rolled-up aggregates (min/hour/day) for 1–3 years, and compressed cold archives for long-term regulatory or root-cause analysis. Make retention policies explicit and automated.

Lineage & provenance

Always attach provenance metadata: source id, capture timestamp, ingestion timestamp, transformation id/version, and owner. Lineage is essential to diagnose why a dashboard value changed and to restore trust after a pipeline change.

Data model & semantics

• Use unique, stable asset IDs and a hierarchical asset model (site → area → line → asset).
• Attach units, precision, and valid ranges for each signal.
• Document whether a stream is sampled, event-driven, or derived and how timestamps are assigned (device vs ingestion time).

Security & compliance

Segment OT/IT networks, enforce least-privilege access, use certificate-based device identities, and limit the exposure of raw OT to central systems. Mask or remove PII before analytics where required by policy.

Alerting & dashboard design principles

1. Design alerts around actions: every alert must answer "what should someone do?" and name the role who should act.
2. Include confidence and context to reduce false positives and unnecessary interruptions.
3. Provide a clear escalation path and attach the runbook or SOP for the alert.
4. On dashboards, show evidence and drilldowns near KPI tiles (last 24h sparkline, recent anomalies, raw signal links).

Common mistakes to avoid

• Embedding opaque business logic in edge filters without recording metadata — makes audits and fixes hard.
• Relying on a single rigid timestamping approach — inconsistent timestamps destroy joins and aggregates.
• Excessive alerting with no owner or runbook — creates alarm fatigue and ignored alerts.
• No catalog or lineage — teams stop trusting dashboards when a value changes and they can’t discover why.

Example technology roles (illustrative)

• Edge runtime / OT gateway: lightweight container or gateway appliance with OPC-UA/MQTT support.
• Message backbone: Kafka or MQTT broker with persistent storage for large-scale, reliable ingest.
• Time-series store: InfluxDB, Timescale, or a managed TSDB for hot queries.
• Data lake / archive: S3 or object store with lifecycle policies.
• Analytics: Spark/Databricks, Flink, or cloud-native streaming tools plus a model-serving layer.
• Alerting & dashboards: Grafana, Kibana, or BI tools with links to runbooks.

Minimal validation checklist (use to quickly assess a design)

• Does each layer have a named owner and SLA?
• Are stable asset IDs and units defined and cataloged?
• Are timestamps and their source (device vs ingestion) recorded consistently?
• Is there a documented retention policy and automated enforcement?
• Are alerts tied to actions, owners, and runbooks?
• Is lineage captured end-to-end and discoverable from dashboards and alerts?
• Are security controls (mutual auth, network segmentation, least privilege) in place?

Next steps and recommended adoption pattern

Start with a single pilot line or site. Implement the minimal pipeline end-to-end (capture → ingestion → TSDB → dashboard → alert) with explicit owners and retention rules. Prove the lineage and runbooks work under simulated failure modes. Then iterate on scale, cataloging, and ML. Capture lessons into a reusable toolkit for other sites.