Skip to main content

tail input

Tails local files with rotation handling and persistent checkpointing.

# Common config fields, showing default values
input:
  label: ""
  tail:
    paths: [] # No default (required)
    exclude: []
    start_at: "end"
    multiline:
      line_start_pattern: ""
      line_end_pattern: ""
    auto_replay_nacks: true

Reads lines from growing files, following them across log rotation. Uses content fingerprinting for file identity and tracks raw byte offsets for correct checkpoint/resume.

Checkpointing is automatic and ack-driven: state is persisted under the edge data directory only when downstream confirms delivery.

Metadata​

This input adds the following metadata fields to each message:

- file_path   Absolute path of the file the line was read from.
- file_name   Base name of the file.

These can be accessed with the metadata("file_path") Bloblang function.

Examples​

Tail Application Logs​

Tail all .log files in a directory, reading only new lines:

input:
  tail:
    paths: [ /var/log/myapp/*.log ]

Tail with Multiline (Java Stack Traces)​

Group multi-line log entries that start with a timestamp:

input:
  tail:
    paths: [ /var/log/myapp/*.log ]
    start_at: beginning
    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2}'

Fields​

paths​

Glob patterns of files to tail (e.g. /var/log/app/*.log).

Type: array of string

exclude​

Glob patterns to exclude from tailing.

Type: array of string
Default: []

start_at​

Where to begin reading when no checkpoint exists for a file.

Type: string
Default: "end"

OptionSummary
beginningRead the file from the start.
endSkip existing content and only read new lines appended after the input starts.

poll_interval​

How often to check for new data and new/rotated files.

Type: string
Default: "200ms"

multiline​

Multiline grouping. Set line_start_pattern to a regex matching the first line of each log entry (e.g. '^\d{4}-\d{2}-\d{2}'). Alternatively, use line_end_pattern for end-of-entry matching. Only one of the two should be set.

Type: object

multiline.line_start_pattern​

Type: string
Default: ""

multiline.line_end_pattern​

Type: string
Default: ""

encoding​

Character encoding of the file (e.g. utf-8, utf-16, latin-1).

Type: string
Default: "utf-8"

max_log_size​

Maximum size of a single log entry. Entries exceeding this size are split.

Type: string
Default: "1MiB"

checkpoint_id​

Explicit checkpoint identity. If set, this value alone determines which checkpoint state is used. Change it to start fresh (e.g. 'v1' → 'v2'). If not set, the checkpoint is derived automatically from the paths.

Type: string
Default: ""

auto_replay_nacks​

Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to false these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation.

Type: bool
Default: true

When to Use​

Unlike tail, the file input is line-streaming but does not survive file rotation or process restarts safely.

Use the tail input when you need to:

  • Tail application or system log files that rotate or roll over by date.
  • Survive edge restarts without data loss — tail resumes from the persisted checkpoint, not from the current end-of-file.
  • Group multi-line entries — Java stack traces, JSON-on-multiple-lines, ASCII-art banners — into single messages with multiline.
  • Read files in non-UTF-8 encodings — latin-1, utf-16, and other text encodings are supported via encoding.
  • Discover new files dropped into a directory at runtime — globs are re-evaluated on every poll_interval.

Don't use this if:

  • You need to read completed files once (load-then-finish), rather than continuously following — use the file input with scanner: lines.
  • The files live on S3 / GCS / Azure — use the cloud-storage inputs (aws_s3, gcp_cloud_storage).
  • You need exactly-once delivery — tail is at-least-once. A crash between the downstream ack and the checkpoint flush will re-deliver the in-flight batch. Plan for downstream idempotence (deduplication key, signed events, an append-only store with a unique constraint).

Common Patterns​

Tail Application Logs​

Watch for new lines continuously, skipping anything written before the input started:

input:
  tail:
    paths: [ /var/log/myapp/*.log ]

The default start_at: end means existing content is skipped on first run; only newly appended lines are forwarded. On every subsequent restart, the persisted checkpoint takes over and reading resumes from the last delivered byte regardless of start_at.

Group Java Stack Traces​

Match the first line of each entry with a timestamp regex:

input:
  tail:
    paths: [ /var/log/myapp/*.log ]
    start_at: beginning
    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2}'

Every line that does not match the pattern is appended to the previous entry. A 50-line stack trace following a timestamped header becomes one message.

Read Multiple Log Sources With One Input​

Globs and explicit paths can be mixed; matched files are deduplicated:

input:
  tail:
    paths:
      - /var/log/nginx/*.log
      - /var/log/myapp/server.log
      - /var/log/myapp/audit-*.log
    exclude:
      - /var/log/nginx/*.gz

Metadata guide​

Every emitted message carries two metadata fields populated from the source file:

Metadata keyDescription
file_pathAbsolute path of the file the line was read from.
file_nameBase name of the file (e.g. server.log).

Reference them with the metadata("file_path") Bloblang function, or with @file_path / @file_name:

pipeline:
  processors:
    - mapping: |
        root.source_file = metadata("file_path")
        root.app          = @file_name.split(".").index(0)   # filename → app id

When the pipeline reads from multiple log sources via one tail, these keys are the primary way to discriminate downstream.

file_path reveals host filesystem layout

file_path is the absolute path on the edge node. Promoting it into the event, as source_file does above, can expose host usernames, mount points, or application directory names to every downstream consumer. Prefer file_name, or map the path to a sanitized identifier, unless consumers genuinely need the full path.

Rotation and file identity​

tail does not match files by name alone. It computes a content fingerprint from each file's first kilobyte, which gives every distinct file a stable identity. The fingerprint is what survives rotation:

  • Rename rotation (server.log → server.log.1, fresh server.log created): the renamed file keeps its identity and is read to its current end; the new server.log becomes a new tracked file at offset 0.
  • Copy-truncate rotation (logrotate copies then truncates): the truncated file is recognized as a new file (its first kilobyte changed) and is read from the start.
  • Inode reuse (the OS reassigns the inode of a deleted file to a new file with the same name): the fingerprint differs from the prior file, so the new file is read from the start without losing offset state for the old one.

Up to three generations of rotated files are tracked at once, so a file that rotated mid-read (server.log → server.log.1 → server.log.2) is still finished cleanly before its state is dropped.

Checkpointing​

tail persists checkpoint state under the edge data directory:

<dataDir>/executions/<pipelineID>/state/tail/<checkpoint-key>/

The checkpoint key is one of:

  • Your checkpoint_id, sanitized for use as a directory name, when you set the field explicitly.
  • Otherwise a key derived from the set of configured paths, with two consequences worth planning for: editing paths changes the key and so starts from a fresh checkpoint, and the key is stable across operating systems, so the same config keeps its checkpoint when it moves between Linux and Windows agents.

When state is written​

Checkpoints are ack-driven, not time-driven. A batch's checkpoint state stays pending until downstream confirms delivery (filtered messages ack normally, so they correctly advance the checkpoint), and the on-disk checkpoint never moves past undelivered data. On a clean shutdown the final poll completes, the queue drains, and the checkpoint reflects everything that was acked. On a crash, the next start re-reads only the un-acked tail.

Checkpoint writes are crash-safe​

A power loss during a checkpoint write yields either the previous checkpoint or the new one, never a half-written file. A genuinely undecodable checkpoint is discarded and the file is re-tailed per start_at; the input does not brick on bad state.

Sharing or resetting state​

  • Share state across config edits. Set checkpoint_id: "logs-v1" explicitly. Adding or removing entries from paths no longer changes the checkpoint, so the input continues from where it left off.
  • Start fresh. Bump the explicit checkpoint_id (e.g. logs-v1 → logs-v2). The new key starts with no state and reads each file from start_at.
  • Migrate to a new pipeline. Checkpoints are anchored under the pipeline ID, so deploying the same tail config under a new pipeline starts fresh by design. Carrying state forward means copying the checkpoint directory from the old pipeline's state path to the new one; that layout is where state happens to live today rather than a stable interface, so prefer the checkpoint_id routes above when they achieve what you need.

Nack and at-least-once delivery​

A message that is nacked (any downstream component reports an error instead of a successful ack) does not advance the checkpoint. The checkpoint never moves past undelivered data — this is the at-least-once contract.

What happens after a nack depends on auto_replay_nacks:

  • true (default). The pipeline runtime redelivers the nacked batch from memory until it acks successfully. The user-visible behavior is: stable backpressure, no data loss, no checkpoint stall.
  • false. The field description above says the rejected messages are deleted, which is the general rule for inputs that keep no position of their own. For tail the deletion is not the end of the story: the batch leaves the pipeline, but the checkpoint does not advance past it, so the next start re-reads it. A permanently rejected batch therefore stalls the checkpoint instead of losing data. Use this when you would rather have a stuck pipeline, which surfaces in monitoring, than keep replaying a batch downstream will never accept. The stall shows up as a non-advancing checkpoint and a growing in-memory queue, since the input keeps one snapshot per un-flushable position.

Consumers should plan for occasional duplicates on restart — a crash between an ack and the checkpoint flush can re-deliver a batch. For deduplication, pair tail with the signature processor and a downstream store keyed by content hash, or with a Bloblang mapping that builds a deterministic event ID from the source file path, line content, and a timestamp.

Multiline grouping​

Many log formats produce entries that span multiple physical lines — Java stack traces, structured-then-formatted JSON, ASCII-art startup banners. multiline reassembles them into one logical message before the line reaches the pipeline.

Set one of the two patterns (the validator rejects both being set):

  • line_start_pattern — regex that matches the first line of each entry. Every subsequent non-matching line is appended to the in-flight entry. Best when entries reliably begin with a timestamp or a level prefix.

    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2}'        # "2026-05-22 ..." starts an entry

  • line_end_pattern — regex that matches the last line of each entry. The line following a match begins a new entry. Useful when entries have a known terminator (e.g. a </request> marker) but no consistent prefix.

    multiline:
      line_end_pattern: '^---END---$'

A trailing partial entry (no terminator yet) is held until the next match arrives, the file rotates, or the input shuts down. On first run with start_at: end, the first multi-line entry may be partial until the next entry begins; switch to start_at: beginning if you need to capture the whole file.

Encoding and large entries​

encoding accepts any IANA-registered text encoding name: utf-8, utf-16, utf-16be, utf-16le, latin-1 / iso-8859-1, windows-1252, gbk, and the like. The encoding is decoded at the byte boundary before lines are split, so multibyte characters that straddle reads are handled correctly. An unsupported value fails at parse time.

max_log_size caps a single logical entry. An entry longer than this is split at the boundary and the remainder is forwarded as a separate message. Tune this for known-bursty entries: 1MiB is fine for most application logs; raise it (10MiB) when JSON-blob log lines or stack-trace dumps can legitimately exceed a megabyte.

  • Worked examples → — five end-to-end pipelines covering a basic tail, multi-line stack traces, JSON-with-fallback parsing, multi-file routing by source, and a hardened production setup with explicit checkpoint identity.
  • file input — load-then-finish file ingestion; use when you don't need rotation handling or checkpoint resume.
  • signature processor — sign every line on its way out, so downstream consumers can verify the line came from the tailing edge node and was not altered.
  • mapping / Bloblang — extract structured fields from log lines, build deterministic event IDs for downstream deduplication.
  • Pipeline error handling — route lines that fail parsing to a dead-letter sink.

Limitations​

  • At-least-once, not exactly-once. A crash between the downstream ack and the checkpoint flush re-delivers the in-flight batch. Plan for downstream idempotence.
  • One writer per checkpoint key. Two tail inputs in the same process pointing at the same checkpoint_id (or, with the implicit key, the same sorted paths) write to the same on-disk state and will race. Give each its own checkpoint_id, or fold them into one input with both globs.
  • Local files only. tail reads from the local filesystem of the edge node. For network shares, mount them locally; for object stores, use the dedicated cloud-storage inputs.
  • No exactly-once tail of remote logs. If you need cryptographic guarantees that every line is delivered exactly once across an edge restart and network partition, combine tail with a downstream content-addressable store and a deduplication processor — neither responsibility belongs to the input itself.