Skip to main content

Send K3s Logs to Multiple Destinations

Send logs to both S3 for long-term storage and OpenSearch for real-time search using the broker output pattern.

Prerequisites and collection scope​

Install kubectl on the edge node and configure a kubeconfig with permission to list pods and read pods/log in production. Replace production and app=web-app below with your namespace and workload label. The command follows the matching pods available when it starts, with at most 10 concurrent log streams; it does not discover new pods continuously. For fleet-wide collection across pod churn, use a Kubernetes log collector. Restarting this command can replay log lines; design downstream storage for duplicates.

These are pipeline configuration fragments. Put input, pipeline, and output under config in a job with name and type: pipeline, as shown in the quickstart.

Pipeline​

input:
  subprocess:
    name: kubectl
    args:
      - logs
      - --all-containers=true
      - --follow
      - --namespace=production
      - --selector=app=web-app
      - --max-log-requests=10
    codec: lines
    restart_on_exit: true

pipeline:
  processors:
    - mapping: |
        root = content().string().parse_json().catch({
          "message": content().string(),
          "level": "info"
        })
        root.node_id = env("NODE_ID")
        root.timestamp = now()

output:
  broker:
    pattern: fan_out
    outputs:
      # Long-term storage in S3
      - aws_s3:
          bucket: edge-k3s-logs-archive
          path: 'logs/${! env("NODE_ID") }/${! timestamp_unix() }-${! uuid_v4() }.jsonl'
          batching:
            count: 5000
            period: 5m
            processors:
              - archive:
                  format: lines

      # Real-time search in OpenSearch
      - opensearch:
          urls: ['https://opensearch.company.com:9200']
          index: 'k3s-logs-${! now().ts_format("2006-01-02") }'
          action: index
          id: '${! uuid_v4() }'
          batching:
            count: 100
            period: 10s

What This Does​

  • Fan-out pattern: Sends each log to both destinations simultaneously
  • S3 for archival: Large batches (5000 logs, 5 minutes) reduce API costs
  • OpenSearch for search: Small batches (100 logs, 10 seconds) enable near-real-time queries
  • JSON parsing: Attempts to parse logs as JSON, falls back to plain text
  • Daily indices: OpenSearch uses date-based indices for easier management

Fan-Out Pattern​

The broker output with pattern: fan_out duplicates each log message and sends it to all configured outputs. Both outputs must succeed for the message to be acknowledged.

Different Batching Strategies​

S3 batching (5000 logs / 5 minutes):

  • Optimized for cost (fewer API calls)
  • Acceptable latency for archival use case

OpenSearch batching (100 logs / 10 seconds):

  • Optimized for freshness (recent logs appear quickly)
  • Higher API call rate acceptable for search use case

Use Cases​

Compliance + operations: Store all logs in S3 for compliance, search recent logs in OpenSearch for debugging

Cost optimization: Keep 7 days in OpenSearch, years in S3

Disaster recovery: If OpenSearch goes down, all logs still flow to S3

Next Steps​