Skip to main content

priority_queue

Durable, priority-aware, disk-bounded store-and-forward buffer.

# Config fields, showing default values
buffer:
  priority_queue:
    priority_field: "priority"
    durability: "crash_safe"
    when_full: "shed"
    max_bytes: ""
    max_delivery_attempts: 10
    ttl: ""
    pre_processors: [] # No default (optional)
    post_processors: [] # No default (optional)

Stores messages on the edge node's local disk before forwarding them downstream. Provides priority-aware delivery (high > normal > low lanes), byte-capped storage with non-silent shedding, age-based TTL expiry, and resumption from disk after a restart.

The buffer acks the input only after the batch is safely stored, so a process crash or restart never loses acknowledged data. Surviving a power cut as well requires durability: power_loss_safe, at a write-throughput cost; the default crash_safe tier leaves a small window (see the durability field).

For throughput, batch at the input: each batch is stored as a single unit, so many tiny batches cost considerably more than one larger batch.

Fields​

priority_field​

Message METADATA field whose value (high|normal|low) selects the lane. This reads metadata, not the payload: set it with a processor before the buffer (input.processors or this buffer's pre_processors), NOT pipeline.processors, which run after the buffer. Records with a missing or unrecognised value use the normal lane.

Type: string
Default: "priority"

durability​

What the buffer is guaranteed to survive. crash_safe (the default) survives every process crash, OOM kill and restart with no loss and never corrupts the spool; a power cut can lose the most recently stored batches. power_loss_safe additionally survives a power cut, at a real write-throughput cost. The tiers are named for what survives rather than for how aggressively they flush, because crash_safe is already safe against everything except losing power mid-write.

Type: string
Default: "crash_safe"

Options: crash_safe, power_loss_safe

when_full​

Policy when max_bytes is reached: shed (drop lowest-priority-oldest first), block (backpressure the input), or reject (error to the input). Has no effect unless max_bytes is set. Note shed only ever drops STRICTLY lower lanes, so on a pipeline where nothing is prioritised it behaves like block.

Type: string
Default: "shed"

Options: shed, block, reject

max_bytes​

Maximum content bytes for this buffer (e.g. 512MiB, 2GiB). Empty = no per-buffer cap; the node-level executor.buffer.max_disk_usage ceiling still applies. This bounds stored content, so the actual disk usage is somewhat higher.

Type: string
Default: ""

max_delivery_attempts​

How many failed deliveries before a batch is treated as undeliverable and PARKED: kept on disk but no longer blocking its lane, and retried periodically with a growing backoff. Failures only count while other deliveries are succeeding, so a destination outage never parks anything. Set 0 to disable parking, in which case an undeliverable batch blocks its lane and every lower-priority lane until it succeeds. Parked data is never deleted by parking itself -- only the same max_bytes and ttl bounds that apply to everything else can remove it.

Type: int
Default: 10

ttl​

Maximum age of a buffered record (e.g. 1h, 24h). Empty = no expiry.

Type: string
Default: ""

pre_processors​

Processors applied to each batch before it is stored to disk.

Type: array of processor

post_processors​

Processors applied to each batch after it is read from disk and before forwarding.

Type: array of processor