Skip to main content

aws_cloudwatch_logs

Consumes log events from AWS CloudWatch Logs.

Polls one or more CloudWatch Log Groups for log events, with optional filtering by log stream names or prefix, a CloudWatch filter pattern, and a configurable start time. Each log event becomes a separate message.

# Config fields, showing default values
input:
  label: ""
  aws_cloudwatch_logs:
    log_group_name: "" # No default (required)
    log_stream_names: []
    log_stream_prefix: ""
    filter_pattern: ""
    start_time: ""
    poll_interval: 5s
    limit: 1000
    structured_log: true
    api_timeout: 30s
    auto_replay_nacks: true
    region: ""
    endpoint: ""
    credentials:
      profile: ""
      id: ""
      secret: ""
      token: ""
      from_ec2_role: false
      role: ""
      role_external_id: ""
    tcp:
      connect_timeout: ""
      keep_alive:
        count: 0
        idle: ""
        interval: ""
      tcp_user_timeout: ""

Examples​

Consume error lines from all streams whose name starts with prod-, beginning at a fixed timestamp.

input:
  aws_cloudwatch_logs:
    log_group_name: my-app-logs
    log_stream_prefix: prod-
    filter_pattern: "[ERROR]"
    start_time: 2024-01-01T00:00:00Z

Fields​

log_group_name​

The name of the CloudWatch Log Group to consume from.

Type: string

log_stream_names​

An optional list of log stream names to consume from. When empty, all streams in the group are consumed.

Type: array
Default: []

log_stream_prefix​

An optional log stream name prefix used to filter which streams are consumed.

Type: string
Default: ""

filter_pattern​

An optional CloudWatch Logs filter pattern to apply when querying log events.

Type: string
Default: ""

start_time​

The time to start consuming log events from. Accepts an RFC 3339 timestamp or the literal now.

Type: string
Default: ""

poll_interval​

The interval at which to poll for new log events.

Type: string
Default: "5s"

limit​

The maximum number of log events to return in a single API call.

Type: int
Default: 1000

structured_log​

Whether to output log events as structured JSON objects with metadata rather than raw strings.

Type: bool
Default: true

api_timeout​

The maximum time to wait for an API request to complete.

Type: string
Default: "30s"

auto_replay_nacks​

Whether rejected messages should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent.

Type: bool
Default: true

region​

The AWS region to target.

Type: string
Default: ""

endpoint​

Allows you to specify a custom endpoint for the AWS API.

Type: string
Default: ""

credentials​

Optional manual configuration of AWS credentials to use. More information can be found in this document.

Type: object

credentials.profile​

A profile from ~/.aws/credentials to use.

Type: string
Default: ""

credentials.id​

The ID of credentials to use.

Type: string
Default: ""

credentials.secret​

The secret for the credentials being used. :::warning Secret This field contains sensitive information that usually shouldn't be added to a config directly, read our secrets page for more info. :::

Type: string
Default: ""

credentials.token​

The token for the credentials being used, required when using short term credentials.

Type: string
Default: ""

credentials.from_ec2_role​

Use the credentials of a host EC2 machine configured to assume an IAM role associated with the instance.

Type: bool
Default: false

credentials.role​

A role ARN to assume.

Type: string
Default: ""

credentials.role_external_id​

An external ID to provide when assuming a role.

Type: string
Default: ""

tcp​

Advanced options for the underlying TCP connection to the AWS API, including connection timeout and keep-alive settings.

Type: object