deadband

A deadband filter (also called report by exception) for repetitive or noisy numeric streams. Per tag, the processor remembers the last value it forwarded and drops any subsequent reading that has not changed meaningfully. The first reading for a tag is always forwarded.

pipeline:
  processors:
    - deadband:
        single:
          value_field: value
          tag_source: meta
          tag_field: node_id
        default:
          mode: absolute
          threshold: 0.5
          heartbeat_interval: 60s

The driving use case is SCADA / IIoT readiness — OPC-UA, Modbus, or Sparkplug B feeds that ship a reading every poll interval whether it changed or not. Nothing about the processor is SCADA-specific: any pipeline carrying numeric values (IoT tags, application gauges, sampled metrics, log-derived counters) can use it.

When to Use

Use the deadband processor when you need to:

• Cut bandwidth and storage on numeric streams where most readings repeat or barely move — sensors, gauges, counters polled on a fixed interval.
• Convert a polled stream to a change-event stream — downstream consumers see one message per meaningful change instead of one per poll.
• Catch fast movements early while still suppressing slow drift — combine a band threshold with a max_slew rate trigger.
• Keep a keep-alive signal so downstream consumers can distinguish unchanged from dead — set heartbeat_interval to forward a reading at least every N seconds even when within the band.
• Tune noise floors per tag — a noisy vibration sensor and a steady temperature sensor get different bands via tag_overrides.

Don't use this if:

• You need to filter on non-numeric content (strings, categories, presence/absence) — use mapping with a meta deleted = ... pattern or a switch processor.
• You need to debounce changes (require N consecutive readings before forwarding) — this is hysteresis, tracked as a follow-up; today the processor forwards the first reading that crosses the band.
• You need to deduplicate identical full messages rather than per-tag numeric readings — use a cache + mapping pattern with a content hash.

Configuration

Common

# Single value per message — the most common SCADA / IIoT shape.
pipeline:
  processors:
    - deadband:
        single:
          value_field: value           # numeric value lives at body.value
          tag_source: meta
          tag_field: node_id           # tag id is the metadata key node_id
        default:
          mode: absolute
          threshold: 0.5               # forward when |new - last| >= 0.5
          heartbeat_interval: 60s      # ...or once a minute regardless

Advanced

# All config fields, showing default values.
pipeline:
  processors:
    - deadband:
        # Exactly one of "single" or "metrics" must be set.
        single:
          value_source: body           # "body" | "meta"
          value_field: value
          tag_source: meta             # "body" | "meta"
          tag_field: node_id

        # ...or, for a message carrying an array of {name, value} metrics:
        # metrics:
        #   path: metrics              # dot-path to the array
        #   name_field: name           # per-element key for the tag id
        #   value_field: value         # per-element key for the value

        default:
          mode: absolute               # "absolute" | "percent"
          threshold: 0                 # 0 forwards on any change (dedup)
          heartbeat_interval: 0s       # 0s disables the keep-alive trigger
          max_slew: 0                  # value units/sec; 0 disables

        tag_overrides:                 # per-tag bands; unset sub-fields inherit default
          vibration-1:
            threshold: 5.0
            heartbeat_interval: 30s
          counter-7:
            mode: percent
            threshold: 1

        on_error: pass                 # "pass" | "drop" | "fail"
        max_tags: 100000               # state cap; 0 = unlimited

Fields

Field	Type	Default	Description
single	object	—	Single-value message shape. Mutually exclusive with metrics; set exactly one.
metrics	object	—	Multi-metric message shape. Mutually exclusive with single; set exactly one.
default	object	—	Band config applied to any tag without a matching per-tag override.
tag_overrides	object map	{}	Per-tag band overrides keyed by tag id. Unset sub-fields inherit the corresponding default field.
on_error	enum	pass	What to do when a reading is missing, non-numeric, or has no tag id. pass logs and forwards the message unchanged; drop discards it; fail returns a processor error so error-handling patterns can route the message.
max_tags	int	100000	Maximum number of distinct tags whose state is tracked. When exceeded, one existing tag is evicted (its next reading is treated as first-seen and forwarded). 0 means unlimited — only safe when the tag set is known to be bounded. Must be ≥ 0.

single sub-fields

Field	Type	Default	Description
value_source	enum	body	Where the numeric value is read from: the JSON body or message metadata.
value_field	string	value	Dot-path into the JSON body (e.g. value, payload.reading) or the metadata key holding the numeric value.
tag_source	enum	meta	Where the tag id is read from: the JSON body or message metadata.
tag_field	string	node_id	Dot-path into the body or the metadata key holding the tag id. The tag id keys per-tag state and per-tag overrides.

metrics sub-fields

Field	Type	Default	Description
path	string	metrics	Dot-path into the JSON body locating the array of metric objects.
name_field	string	name	Key within each metric object holding the tag id.
value_field	string	value	Key within each metric object holding the numeric value.

default and tag_overrides band sub-fields

Field	Type	Default	Description
mode	enum	absolute	How the band is measured. absolute: forward when `
threshold	float	0	Band width. 0 forwards on any change (consecutive identical values are deduped). Must be ≥ 0.
heartbeat_interval	duration	0s	Keep-alive interval. If > 0s, forward a reading when this much time has passed since the last forward for that tag, even when the value is within the band. 0s disables.
max_slew	float	0	Rate-of-change trigger in value units per second. If > 0, forward when `

The submission validator rejects unknown enum values, negative threshold / max_slew / max_tags, and surfaces a warning when a percent threshold is > 100 (likely intended as a fraction — 0.5 written instead of 50). Identical numeric checks run again at processor construction, so hand-built configs cannot bypass them.

How forwarding works

For each (tag, value), the processor evaluates the configured triggers in order and forwards the reading if any of them fires:

1. First reading for the tag. A tag with no prior state is always forwarded; its value becomes the baseline.
2. Heartbeat elapsed. If heartbeat_interval > 0 and at least that long has passed since the tag was last forwarded, the reading is forwarded — even when the value is within the band. The new value becomes the baseline.
3. Rate of change exceeds max_slew. If max_slew > 0, |new − last| / dt (in value units per second, since the last forward) at or above max_slew forwards the reading. Catches a fast ramp before the band would catch it.
4. Band exceeded. Absolute: |new − last| ≥ threshold. Percent: |new − last| ≥ |last| × threshold / 100. A delta of exactly zero is never meaningful — identical consecutive values are always deduped.

Important details verified from the runtime:

• Comparisons are against the last forwarded value, not the last seen value. Slow sub-threshold drift therefore accumulates and eventually crosses the band, instead of being suppressed forever.
• Dropped readings do not update state. Only a forwarded reading resets the baseline value and timestamp.
• Backward clock steps are safe. A negative dt makes the heartbeat check false and skips the slew check; only the band test applies until the clock recovers.
• Non-finite values are rejected. NaN, +Inf, -Inf, and out-of-range JSON numbers route through on_error rather than poisoning the baseline. (strconv.ParseFloat("NaN") succeeds without error in Go — without this guard one bad reading would silently disable deadbanding for a tag forever.)

Source shapes

single — one reading per message

The whole message is forwarded or dropped. This is the SCADA/IIoT "one tag per poll" shape — OPC-UA, Modbus, and most CSV-over-Kafka feeds emit messages like this. Tag ids typically come from message metadata (set by an earlier mapping step or by the input itself), and the numeric value lives at a known body path.

- deadband:
    single:
      value_field: value         # body.value
      tag_source: meta
      tag_field: node_id         # metadata.node_id
    default:
      mode: absolute
      threshold: 0.5

Either field can be read from the body or from metadata — set value_source / tag_source independently. A numeric tag id (e.g. a JSON number) is coerced to its string form so it can key the state map.

metrics — array of metrics per message

A message carries an array of {name, value, ...} objects (Sparkplug B-style birth/death events, OpenTelemetry-style metric snapshots, batched gauge updates). Each metric is deadbanded independently. The output payload preserves the surrounding JSON shape but contains only metrics whose readings crossed their band; if every metric in the array was within its band, the whole message is dropped.

- deadband:
    metrics:
      path: metrics              # body.metrics is the array
      name_field: name
      value_field: value
    default:
      mode: absolute
      threshold: 1.0

Passthrough byte preservation. When no metric was dropped (every reading in the batch crossed its band), the processor forwards the original message bytes untouched rather than re-marshaling. This is deliberate: Go's encoding/json reorders object keys on re-marshal, which silently rewrites the payload and breaks downstream signing or any order-sensitive consumer.

When the array is partially filtered, the surrounding object is re-encoded with the filtered array spliced back in at path. If you need byte-stable output even for partial filtering, place the signature processor after deadband so the signature is computed over the canonical filtered form.

Other paths

Both shapes accept dot-paths into nested JSON:

# Single value at a nested path:
- deadband:
    single:
      value_field: payload.reading
      tag_source: meta
      tag_field: sensor_id

# Metrics array at a nested path:
- deadband:
    metrics:
      path: payload.metrics
      name_field: name
      value_field: value

Paths address only objects at intermediate steps — an intermediate array or scalar produces a "not found" error that routes through on_error.

Band tuning

absolute mode

absolute is the default. Forward when |new − last| ≥ threshold. Best when the signal has a roughly known scale and a fixed noise floor — a temperature sensor reading °C with ±0.3°C jitter, set threshold: 0.5.

default:
  mode: absolute
  threshold: 0.5

percent mode

Forward when |new − last| ≥ |last| × threshold / 100. Useful when a signal spans many orders of magnitude (flow rates, pressures, counts) — a 1% band is meaningful at both 100 and 10 000.

default:
  mode: percent
  threshold: 5     # 5%

Percent thresholds are 0–100, not 0–1

The submission validator emits a warning when a percent threshold is > 100 because that almost always means the author wrote a fraction. 0.5 here means 0.5%, not 50%. To get 50%, write 50.

heartbeat_interval — distinguish dead from unchanged

A signal that has stopped reporting and a signal that is reporting the same value every poll look identical downstream once deadbanding drops the duplicates. Set heartbeat_interval to forward at least one reading per interval per tag regardless of value movement, so consumers can tell unchanged from dead.

default:
  mode: absolute
  threshold: 0.5
  heartbeat_interval: 60s

A heartbeat-driven forward resets the baseline to the current value. Subsequent band comparisons measure against that last emitted sample, not against the last sample that crossed the band on its own.

max_slew — catch fast movements early

A wide band suppresses noise but also delays detection of a real ramp. max_slew adds an orthogonal trigger: forward when the rate of change since the last forward exceeds the configured rate, in value units per second.

default:
  mode: absolute
  threshold: 10              # noise floor
  max_slew: 2.0              # ...but forward immediately on > 2 units/sec

max_slew is in value units per second, not per message. The rate is |new − last| / dt, where dt is wall-clock seconds since the last forward for that tag.

Per-tag overrides

tag_overrides is an object map keyed by tag id. Each entry is a partial band config; any sub-field the override does not set inherits from default.

default:
  mode: absolute
  threshold: 0.5
  heartbeat_interval: 60s
tag_overrides:
  vibration-1:               # noisier sensor — wider band, faster heartbeat
    threshold: 5.0
    heartbeat_interval: 30s
  counter-7:                 # this tag uses percent mode instead
    mode: percent
    threshold: 1
  pressure-9:                # only the slew differs
    max_slew: 50

Overrides are resolved at processor construction. A tag_overrides entry with no recognized sub-fields is equivalent to no override at all for that tag.

Error handling

on_error controls what happens when a reading is missing, non-numeric, has no tag id, or the body cannot be parsed as a JSON object:

Value	Behavior
pass (default)	Log the error and forward the message unchanged.
drop	Log the error and silently drop the message (or the offending metric, in metrics mode).
fail	Return a processor error so the message is routed by error handling (e.g. to a dead-letter queue).

In metrics mode, on_error applies per metric: a single malformed metric in a batch follows the configured posture, while the surrounding message and the rest of the batch are still processed normally.

State bounds

The processor keeps one state entry per distinct tag id seen. With a bounded physical tag set (the typical SCADA case), this is naturally small — a few hundred to a few thousand entries — and max_tags rarely matters. But when tag_source: body or metrics mode is used, the tag id comes from message content; a high-cardinality stream (or an adversarial one) could otherwise grow state without bound.

max_tags caps the state map:

• Default 100000 — a generous backstop for almost any real workload.
• 0 means unlimited. Only safe when the tag set is known to be bounded.
• When the cap is reached, one existing tag is evicted before a new one is tracked. Eviction is O(1) (an arbitrary entry, not strict LRU). An evicted tag's next reading is treated as first-seen and forwarded — the conservative choice. The processor logs one WARN the first time it evicts, with the configured cap; subsequent evictions are silent.

If you see the eviction warning, either raise max_tags or use a more constrained tag source (move tag ids into metadata that you set, rather than reading them from untrusted body content).

Worked example

A sensor reports temperature once a second. Below, with threshold: 0.5 and heartbeat_interval: 60s:

Time (s)	Reading	Forwarded?	Why
0	21.5	✅	First reading
1	21.5	❌	Δ = 0, identical
2	21.7	❌	Δ = 0.2 < 0.5
3	21.8	❌	Δ = 0.3 < 0.5 (still measuring against last forwarded 21.5)
4	22.0	✅	Δ = 0.5 ≥ 0.5 — new baseline is 22.0
5–63	22.0 ± 0.1	❌	All within [21.5, 22.5]
64	22.1	✅	Heartbeat: 60s since last forward — new baseline is 22.1
65	22.1	❌	Δ = 0

Notice the drift catch at t=4: even though no single reading moved by 0.5, the cumulative drift from the last forwarded value did, so the band fired. This is the property that makes deadbanding safe — slow drift is not suppressed forever.

Related

• Worked examples → — five end-to-end recipes: simple absolute band, Sparkplug B metrics array, percent band for wide-magnitude signals, ramp detection via max_slew, and a SCADA-shaped pipeline with per-tag overrides.
• signature processor — pair after deadband so signatures cover the deadbanded output, not the dropped originals.
• metadata processor — set tag ids in metadata declaratively so deadband can read them via tag_source: meta.
• Pipeline error handling — route messages that on_error: fail rejects to a dead-letter sink.
• mapping / Bloblang guide — when a band rule is not enough and you need conditional or computed filtering.

Limitations

• No hysteresis / debounce. A reading that just crosses the band is forwarded immediately; there is no "must hold for N seconds" filter. Tracked as a follow-up.
• State is per-process, in-memory. Restarting the edge resets every tag's baseline; the next reading per tag will be forwarded as first-seen. The processor does not currently persist state across restarts.
• Eviction is O(1), not strict LRU. When max_tags is exceeded, the evicted tag is arbitrary, not least-recently-used.
• Wall-clock heartbeat / slew. Heartbeats and slew rates are measured against the edge node's wall clock, not against the timestamps inside messages. Replay scenarios that need message-time semantics are tracked as a follow-up.
• Numeric values only. Strings, booleans, and structured types are not numeric and route through on_error. Numeric strings ("21.5") read from metadata are parsed; numeric strings in the body are not.