Deadband Filtering for Numeric Streams
This page collects five end-to-end pipelines that use the deadband processor to drop noisy or unchanged numeric readings before they leave the edge. Each example is a working pipeline you can run as-is; the surrounding inputs and outputs are kept minimal so the focus stays on the deadband configuration.
For the full configuration surface, see the deadband processor reference.
1. Simple absolute band with heartbeat
A SCADA tag streams a reading every second. We only want to forward a reading when the value has moved by at least 0.5 units, or once a minute regardless so downstream can tell unchanged from dead.
input:
generate:
interval: 1s
mapping: |
root.value = random_int(min: 200, max: 220) * 0.1
meta node_id = "sensor-1"
pipeline:
processors:
- deadband:
single:
value_field: value # body.value
tag_source: meta
tag_field: node_id # metadata.node_id
default:
mode: absolute
threshold: 0.5
heartbeat_interval: 60s
output:
stdout: {}
What survives. With a synthetic stream centered around 21.0 ± 1.0, you will see one reading at startup (the first reading is always forwarded), then a forward each time the value drifts more than 0.5 units from the last forwarded baseline, plus one keep-alive every 60 seconds even when nothing has moved.
Why tag_source: meta. Tag ids do not change message-to-message for a given sensor — they belong on the message envelope, not in the body. Reading from metadata also lets you set the tag id with a single mapping step at ingest and never repeat it in the payload.
2. Sparkplug-style metrics array
Many industrial protocols ship a batch of metrics in a single message — a Sparkplug B device data event, an OpenTelemetry metric snapshot, an MQTT payload with several gauges. The metrics source shape deadbands each metric independently and rewrites the array with the sub-threshold metrics removed.
input:
generate:
interval: 5s
mapping: |
root.device = "plc-12"
root.metrics = [
{ "name": "rpm", "value": 1500 + random_int(min: -2, max: 2) },
{ "name": "temp", "value": 40.0 + random_int(min: -1, max: 1) * 0.05 },
{ "name": "psi", "value": 75.0 }
]
pipeline:
processors:
- deadband:
metrics:
path: metrics # body.metrics
name_field: name # per-element tag id
value_field: value
default:
mode: absolute
threshold: 1.0
output:
stdout: {}
What survives. rpm swings by 2 units between messages, so it crosses the band on most ticks. temp moves by 0.05 each tick, well under the band, so it is filtered out of the array. psi is constant, so after the first message it disappears from the output array entirely. When every metric falls within the band on a given tick, the whole message is dropped; when some survive, the output JSON keeps the surrounding device field and contains only the surviving metrics under metrics.
Byte-stable passthrough. If no metric is filtered on a given message — every tag crossed its band that tick — the processor forwards the original bytes untouched, not a re-marshaled copy. This matters if you sign messages downstream: a re-marshal would reorder JSON keys and break the signature. If you sign after filtering, place the signature processor after deadband so it operates on the canonical filtered form.
3. Percent band for wide-magnitude signals
A flow-rate sensor reads anywhere from 5 to 5000 units per second depending on plant state. A fixed absolute band has to be tuned to the peak magnitude, which then suppresses meaningful changes at the low end. A percent band scales naturally: a 5% move is meaningful at both 50 and 5000.
input:
generate:
interval: 1s
mapping: |
root.reading = random_int(min: 100, max: 5000)
meta tag = "flow-3"
pipeline:
processors:
- deadband:
single:
value_field: reading
tag_source: meta
tag_field: tag
default:
mode: percent
threshold: 5 # forward on a 5% move
output:
stdout: {}
The submission validator emits a warning when a percent threshold is > 100, because it almost always means the author wrote a fraction. 0.5 here would mean 0.5%, not 50%. To get 50%, write 50.
4. Catch fast ramps with max_slew
A pressure signal sits at 75 PSI with ±1 PSI jitter, which an absolute band of 5 PSI happily suppresses. But when a leak develops the value ramps from 75 to 60 PSI over 4 seconds — by the time the band fires you have already missed the early part of the event. Adding a max_slew trigger forwards a reading as soon as the rate of change crosses a threshold, even if the band would still suppress it.
input:
generate:
interval: 1s
mapping: |
# 0.8 chance of jitter, 0.2 chance of a step toward 60.
root.value = if random_int(min: 0, max: 9) < 8 {
75.0 + (random_int(min: -10, max: 10) * 0.1)
} else {
75.0 - random_int(min: 0, max: 15)
}
meta tag_id = "pressure-1"
pipeline:
processors:
- deadband:
single:
value_field: value
tag_source: meta
tag_field: tag_id
default:
mode: absolute
threshold: 5 # noise floor — wide enough to swallow jitter
max_slew: 2.0 # but forward when |Δ| / Δt ≥ 2 PSI/s
output:
stdout: {}
max_slew is measured in value units per second, where seconds are wall-clock time since the last forward for that tag. The band and the slew triggers are independent: a forward fires when any of first-seen, heartbeat, slew, or band is true. A dt of zero or negative (a backward clock step) is skipped safely — only the band test applies until the clock recovers.
5. SCADA-shaped pipeline with per-tag overrides
A real SCADA / IIoT deployment runs a heterogeneous tag set on one pipeline: a noisy vibration sensor, a slow temperature signal, a high-rate-of-change pressure, and a low-cardinality counter. Each needs its own tuning. The default block sets the floor; tag_overrides patches individual tags.
input:
kafka:
addresses: [ "broker:9092" ]
topics: [ "scada.tags" ]
consumer_group: "deadband-edge"
pipeline:
processors:
# Earlier steps will have set metadata.sensor_id and shaped the body to
# { reading: <number> }. The deadband only forwards a meaningful change
# per sensor — every other reading is dropped here, at the edge.
- deadband:
single:
value_source: body
value_field: reading
tag_source: meta
tag_field: sensor_id
default:
mode: absolute
threshold: 0.5
heartbeat_interval: 60s # one keep-alive per tag per minute
tag_overrides:
vibration-1: # noisy mechanical signal: wider band, faster heartbeat
threshold: 5.0
heartbeat_interval: 30s
pressure-9: # mostly stable but occasional fast ramps
threshold: 2.0
max_slew: 1.5
counter-7: # high-magnitude monotonic counter — use percent
mode: percent
threshold: 1
on_error: drop # discard malformed readings rather than passing them through
max_tags: 50000 # bound state when sensor_id comes from message content
output:
kafka:
addresses: [ "broker:9092" ]
topic: "scada.tags.deadbanded"
Why these settings.
tag_source: metakeeps the tag id on the envelope so a content-driven payload cannot create a new tracked tag —max_tagsis then a memory backstop, not a routine control.- Default
heartbeat_interval: 60spreserves a keep-alive signal per tag even when nothing has moved, so downstream consumers can tell quiet from dead. - Default
threshold: 0.5is the noise floor for the majority of tags; only the outliers (vibration-1,pressure-9,counter-7) need overrides. on_error: dropis appropriate when malformed readings are non-recoverable noise. Usefailinstead to route them through error handling into a dead-letter topic.
Compounding with downstream signing. Adding a signature processor after deadband produces signed, deadbanded events: downstream consumers can verify both that the reading came from the producing edge node and that the bytes were not altered after filtering. The signature covers the filtered output, not the dropped originals — which is the right behavior for audit and compliance use cases.
Common variations
Read the value from metadata too
When an upstream step has already pulled the numeric value into metadata (a parser, an enrichment step, or an OPC-UA source), point value_source at meta:
- deadband:
single:
value_source: meta
value_field: reading # metadata.reading
tag_source: meta
tag_field: sensor_id
default:
mode: absolute
threshold: 0.5
Metadata values are parsed with strconv.ParseFloat. NaN, +Inf, -Inf, and overflowing values are rejected (they would otherwise silently disable deadbanding for the tag) and route through on_error.
Forward on any non-zero change (deduplication only)
A threshold: 0 band forwards on any change — useful for deduplicating identical consecutive readings without further filtering. Combine with heartbeat_interval to get a "one reading per change, plus a keep-alive" stream.
default:
mode: absolute
threshold: 0
heartbeat_interval: 5m
Inherit only the parts you want to override
tag_overrides entries are partial; any sub-field you do not set inherits from default. To run one tag in percent mode while everything else stays absolute, set only mode and threshold on the override:
default:
mode: absolute
threshold: 0.5
heartbeat_interval: 60s # inherited by every override below
tag_overrides:
counter-7:
mode: percent
threshold: 1 # 1%; heartbeat_interval still 60s
vibration-1:
threshold: 5.0 # absolute mode (inherited); just a wider band
Next Steps
deadbandprocessor reference — every field, every default, the decision logic, error postures, and state bounds.signatureprocessor — sign the deadbanded output for tamper-evident audit trails.- Pipeline error handling — route messages that fail validation to a dead-letter sink.
- Quick Start — how to actually deploy and run these pipelines on an edge node.