Skip to main content

file input

Consumes data from files on disk, emitting messages according to a chosen codec.

# Common config fields, showing default values
input:
  label: ""
  file:
    paths: [] # No default (required)
    scanner: {"lines":{}}
    auto_replay_nacks: true

Metadata​

This input adds the following metadata fields to each message:

- path
- mod_time_unix
- mod_time (RFC3339)

You can access these metadata fields using function interpolation.

Examples​

Read a Bunch of CSVs​

If we wished to consume a directory of CSV files as structured documents we can use a glob pattern and the csv scanner:

input:
  file:
    paths: [ ./data/*.csv ]
    scanner:
      csv: {}

Fields​

paths​

A list of paths to consume sequentially. Glob patterns are supported, including super globs (double star).

Type: array of string

scanner​

The scanner by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the csv scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once.

Type: scanner
Default: {"lines":{}}

delete_on_finish​

Whether to delete input files from the disk once they are fully consumed.

Type: bool
Default: false

auto_replay_nacks​

Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to false these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation.

Type: bool
Default: true

When to Use​

Use file input when you need to:

  • Read log files from /var/log/ or application directories
  • Process CSV/JSON files dropped into a directory

Don't use this if:

  • You need continuous tailing — file reads each matched file to completion and stops; it does not follow rotated files and does not resume after a process restart. Use tail, which survives rotation and restarts via persisted checkpoints.
  • Files are on S3/GCS — use aws_s3 or gcp_cloud_storage
  • You need stdin — use stdin

Common Patterns​

Read Log Files Line by Line​

Read the existing contents of matched files, one message per line, then finish. For following logs as they grow, use tail instead:

input:
  file:
    paths: ["/var/log/app/*.log"]
    scanner:
      lines: {}

Process All JSON Files​

Read complete files as documents:

input:
  file:
    paths: ["/data/incoming/*.json"]
    scanner:
      json_documents: {}

CSV Processing​

Parse CSV with headers:

input:
  file:
    paths: ["/data/*.csv"]
    scanner:
      csv: {}