Skip to main content

IoT Data Aggregation

Overview​

IoT deployments generate massive volumes of high-frequency data from sensors, devices, and equipment. Sending every reading to the cloud creates extreme bandwidth costs, storage challenges, and delayed anomaly detection. Processing data at the edge - aggregating, downsampling, and analyzing locally before transmission - dramatically reduces costs while improving real-time responsiveness.

Expanso's Approach to IoT Data Aggregation​

Expanso Edge processes sensor data at the source, running on edge gateways or industrial PCs near your IoT devices. Agents receive high-frequency readings via MQTT or HTTP, perform aggregation and analysis locally, and send only meaningful insights to cloud systems.

Key capabilities:

  • Time-Window Aggregation: Collect high-frequency readings (per-second) and aggregate into configurable intervals (per-minute, per-hour), calculating statistics like mean, min, max, and percentiles.
  • Anomaly Detection: Identify outliers, sudden changes, and pattern deviations in real-time at the edge, triggering immediate alerts before data reaches the cloud.
  • Downsampling with Fidelity: Send fewer records through statistical summaries and edge-based filtering. Choose the statistics needed by downstream consumers; summaries cannot preserve every raw reading.
  • Protocol Flexibility: Ingest data from MQTT brokers or HTTP endpoints, all in a single pipeline.
  • Offline Resilience: Configure buffering and retry behavior for outages. Retention depends on buffer capacity, persistence, source acknowledgements, and outage duration.

Benefits of Edge IoT Processing​

Processing IoT data at the edge provides significant advantages:

Cost Reduction

  • Reduce bandwidth use when summaries are smaller than the raw readings they replace
  • Lower cloud ingestion costs by sending summaries instead of raw readings
  • Minimize storage costs with pre-filtered, aggregated datasets
  • Measure ingest, storage, edge compute, and network costs together to estimate savings for your workload

Real-Time Operations

  • Sub-second anomaly detection and alerting at the source
  • Local dashboards provide instant visibility without cloud latency
  • Immediate automated responses to critical sensor readings
  • Maintain operations during network outages with local processing

Data Quality

  • Preserve analytical fidelity through statistical aggregation (mean, percentiles, variance)
  • Use appropriately sized durable buffers to reduce loss during downstream outages
  • Monitor gaps and buffer exhaustion during network interruptions
  • Keep processing in selected regions as one part of your data sovereignty controls

Worked aggregation example​

A sensor reporting once per second produces 60 records per minute. Replacing those records with one minute-level summary reduces the record count by 59 / 60, or about 98.3%. This is an illustrative record-count calculation, not a measured bandwidth or cost result: summary size, metadata, compression, retries, and edge compute affect the actual savings. Keep raw readings when investigations or downstream calculations require them.

Common Patterns​

Statistical Aggregation Collect per-second sensor readings and aggregate into minute or hour intervals. Calculate mean, min, max, standard deviation, and percentiles according to the analytical requirements of downstream consumers.

Threshold-Based Alerting Monitor sensor readings against configurable thresholds at the edge. Trigger immediate alerts when values exceed limits, fall outside normal ranges, or show sudden rate-of-change anomalies.

Multi-Protocol Collection Combine data from MQTT sensors and HTTP devices in a single pipeline. Normalize formats and enrich with location metadata before aggregation.

Tiered Storage Send aggregated summaries to cloud analytics platforms every few minutes, while keeping high-frequency raw data locally for short-term analysis and backfill operations.

Edge Dashboards Serve real-time operational dashboards directly from edge locations using local Grafana or custom HTTP endpoints, eliminating cloud round-trips for monitoring.

Example Use Cases​

  • Manufacturing plants aggregating sensor data from thousands of machines, detecting equipment anomalies locally, and sending only summaries to centralized analytics
  • Smart buildings processing HVAC, occupancy, and energy sensors in real-time, optimizing operations locally while reporting trends to cloud systems
  • Agriculture deployments monitoring soil moisture, temperature, and weather data across remote farms with intermittent connectivity
  • Energy infrastructure collecting high-frequency telemetry from solar arrays, wind turbines, and grid equipment, detecting faults immediately while archiving aggregated performance data

Production Examples​

Data TransformationIntermediate

Aggregate Time Windows

Windowing, aggregations, and metrics for high-frequency sensor data.

View example →
Data RoutingIntermediate

Smart Buffering

Backpressure and buffer management for reliable data delivery.

View example →
Data TransformationBeginner

Deduplicate Events

Event deduplication and idempotency for clean data streams.

View example →
Data TransformationBeginner

Normalize Timestamps

Time zone handling and ISO 8601 normalization.

View example →
IntegrationAdvanced

O-RAN Telco Pipeline

Multi-destination telemetry for OpenShift edge deployments.

View example →

Next Steps​