Skip to main content

Filter K3s Logs by Level

Reduce log volume and storage costs by filtering to only ERROR and WARN level logs.

Pipeline​

input:
  subprocess:
    name: kubectl
    args:
      - logs
      - --all-containers=true
      - --prefix=true
      - --follow
      - --all-namespaces
    codec: lines
    restart_on_exit: true

pipeline:
  processors:
    - mapping: |
        # Parse JSON logs if possible
        root = this.parse_json().catch({
          "message": this,
          "level": "info"
        })
        root.timestamp = now()
        root.node_id = env("NODE_ID")

    # Only keep ERROR and WARN logs
    - switch:
        - check: |
            ["error", "warn", "fatal"].any(t -> this.level.or("").lowercase().contains(t))
          processors:
            - mapping: 'root = this'

output:
  aws_s3:
    bucket: edge-k3s-errors
    path: 'errors/${! env("NODE_ID") }/${! timestamp_unix() }.json'
    batching:
      count: 100
      period: 1m

What This Does​

  • Parses JSON logs: Attempts to extract level field from JSON-formatted logs
  • Filters by level: Only passes through logs containing "error", "warn", or "fatal"
  • Drops other logs: INFO and DEBUG logs are discarded
  • Smaller batches: 100 logs since error volume is much lower

Volume Reduction​

Typical production applications generate:

  • 90% INFO logs
  • 8% DEBUG logs
  • 2% WARN/ERROR/FATAL logs

This filter reduces log volume by ~90% while retaining all error information.

Handling Non-JSON Logs​

The parse_json().catch() pattern handles both JSON and plain text logs:

JSON log (parsed):

{"level": "ERROR", "message": "Database connection failed"}

Plain text log (fallback):

[2024-11-09] ERROR: Database connection failed

For plain text logs, the default level is "info", so they're filtered out unless they contain error keywords in the message.

Keyword Matching​

The filter uses any() plus contains() to match multiple error levels:

["error", "warn", "fatal"].any(t -> this.level.or("").lowercase().contains(t))

Matches: error, Error, ERROR, warning, WARN, fatal, FATAL

Cost Impact​

Before filtering (10,000 logs/minute):

  • S3 storage: ~50 GB/day
  • Elasticsearch: ~150 GB/month

After filtering (1,000 logs/minute):

  • S3 storage: ~5 GB/day
  • Elasticsearch: ~15 GB/month

Savings: ~90% reduction in storage and processing costs

Next Steps​