Skip to main content

Collect Logs from Specific Namespace

Collect logs from a single namespace instead of all namespaces to reduce volume and focus on specific applications.

Pipeline​

input:
  subprocess:
    name: kubectl
    args:
      - logs
      - --namespace=production
      - --all-containers=true
      - --prefix=true
      - --follow
    codec: lines
    restart_on_exit: true

pipeline:
  processors:
    - mapping: |
        root.log = this
        root.namespace = "production"
        root.node_id = env("NODE_ID")
        root.timestamp = now()

output:
  http_client:
    url: https://logs.company.com/ingest
    verb: POST
    batching:
      count: 500
      period: 30s

What This Does​

  • Namespace filtering: Only collects logs from the production namespace
  • Reduced volume: Ignores logs from other namespaces (kube-system, monitoring, etc.)
  • HTTP output: Sends logs to a custom log ingestion endpoint
  • Smaller batches: 500 logs or 30 seconds for faster delivery

Use Cases​

Production monitoring: Only collect logs from production workloads, ignore system pods

Multi-tenant clusters: Separate log collection per tenant namespace

High-volume namespaces: Isolate logs from specific high-traffic applications

Compliance: Collect logs only from namespaces with compliance requirements

Multiple Namespace Pipelines​

Run multiple Expanso pipelines to collect from different namespaces:

production-logs.yaml:

input:
  subprocess:
    name: kubectl
    args: [logs, --namespace=production, --follow]
output:
  aws_s3:
    bucket: production-logs

staging-logs.yaml:

input:
  subprocess:
    name: kubectl
    args: [logs, --namespace=staging, --follow]
output:
  aws_s3:
    bucket: staging-logs

Run both:

expanso-edge run --config production-logs.yaml &
expanso-edge run --config staging-logs.yaml &

Namespace Patterns​

Collect from multiple specific namespaces: Run separate pipelines for each

Exclude system namespaces: Use --all-namespaces and filter out kube-system, kube-public

Dynamic namespace selection: Use environment variables:

args:
  - logs
  - --namespace=${NAMESPACE}
  - --follow

Next Steps​