Skip to main content

Filter K3s Logs by Level

Reduce log volume and storage costs by filtering to only ERROR and WARN level logs.

Prerequisites and collection scope​

Install kubectl on the edge node and configure a kubeconfig with permission to list pods and read pods/log in production. Replace production and app=web-app below with your namespace and workload label. The command follows the matching pods available when it starts, with at most 10 concurrent log streams; it does not discover new pods continuously. For fleet-wide collection across pod churn, use a Kubernetes log collector. Restarting this command can replay log lines; design downstream storage for duplicates.

These are pipeline configuration fragments. Put input, pipeline, and output under config in a job with name and type: pipeline, as shown in the quickstart.

Pipeline​

input:
  subprocess:
    name: kubectl
    args:
      - logs
      - --all-containers=true
      - --follow
      - --namespace=production
      - --selector=app=web-app
      - --max-log-requests=10
    codec: lines
    restart_on_exit: true

pipeline:
  processors:
    - mapping: |
        # Parse JSON logs if possible
        root = content().string().parse_json().catch({
          "message": content().string(),
          "level": "info"
        })
        root.timestamp = now()
        root.node_id = env("NODE_ID")

    # Only keep ERROR and WARN logs
    - mapping: |
        root = if ["error", "warn", "warning", "fatal"].contains(this.level.or("").lowercase()) { this } else { deleted() }

output:
  aws_s3:
    bucket: edge-k3s-errors
    path: 'errors/${! env("NODE_ID") }/${! timestamp_unix() }-${! uuid_v4() }.jsonl'
    batching:
      count: 100
      period: 1m
      processors:
        - archive:
            format: lines

What This Does​

  • Parses JSON logs: Attempts to extract level field from JSON-formatted logs
  • Filters by level: Only passes through records whose level equals "error", "warn", "warning", or "fatal", ignoring case
  • Drops other logs: INFO and DEBUG logs are discarded
  • Smaller batches: 100 logs since error volume is much lower

Volume Reduction​

Reduction depends on your log mix. If 2% of records have one of the selected levels, this filter retains 2% and discards 98% of records. This is illustrative, not a measured byte or storage-cost saving. Review the discarded levels before using this for production diagnostics.

Handling Non-JSON Logs​

The parse_json().catch() pattern handles both JSON and plain text logs:

JSON log (parsed):

{"level": "ERROR", "message": "Database connection failed"}

Plain text log (fallback):

[2024-11-09] ERROR: Database connection failed

For plain text logs, the default level is "info", so they are discarded even when their message includes an error keyword. Add a parser for your text format before filtering if you need to retain those errors.

Level Matching​

The filter compares the lowercase level field against error, warn, warning, and fatal. It does not search the message text. deleted() explicitly discards every other record.

Next Steps​