Skip to main content

Buffers

A buffer is a mechanism for storing messages temporarily between your input and output, which can help with performance and reliability:

input:
  kafka:
    addresses: [ localhost:9092 ]
    topics: [ events ]

buffer:
  memory:
    limit: 10000

output:
  aws_s3:
    bucket: my-bucket
    path: 'data/${! timestamp_unix() }.json'

Buffers are optional. When configured, they allow the input to continue receiving messages even when the output is temporarily slower or unavailable, preventing message loss during backpressure.

When to Use Buffers​

Use buffers when:

  • Your output is slower than your input
  • You experience temporary network issues
  • You want to smooth out traffic bursts
  • You need durability across restarts (with sqlite buffer)

For most pipelines where input and output rates are matched, you can use the default none buffer.