Skip to main content

Collect Logs from Specific Namespace

Collect logs from a labeled workload in one namespace to reduce volume and focus on specific applications.

Prerequisites and collection scope​

Install kubectl on the edge node and configure a kubeconfig with permission to list pods and read pods/log in production. Replace production and app=web-app below with your namespace and workload label. The command follows the matching pods available when it starts, with at most 10 concurrent log streams; it does not discover new pods continuously. For fleet-wide collection across pod churn, use a Kubernetes log collector. Restarting this command can replay log lines; design downstream storage for duplicates.

These are pipeline configuration fragments. Put input, pipeline, and output under config in a job with name and type: pipeline, as shown in the quickstart.

Pipeline​

input:
  subprocess:
    name: kubectl
    args:
      - logs
      - --namespace=production
      - --selector=app=web-app
      - --max-log-requests=10
      - --all-containers=true
      - --prefix=true
      - --follow
    codec: lines
    restart_on_exit: true

pipeline:
  processors:
    - mapping: |
        root.log = content().string()
        root.namespace = "production"
        root.node_id = env("NODE_ID")
        root.timestamp = now()

output:
  http_client:
    url: https://logs.company.com/ingest
    verb: POST
    batching:
      count: 500
      period: 30s

What This Does​

  • Namespace filtering: Collects only pods matching app=web-app in the production namespace
  • Reduced volume: Ignores logs from other namespaces (kube-system, monitoring, etc.)
  • HTTP output: Sends logs to a custom log ingestion endpoint
  • Smaller batches: 500 logs or 30 seconds for faster delivery

Use Cases​

Production monitoring: Only collect logs from production workloads, ignore system pods

Multi-tenant clusters: Separate log collection per tenant namespace

High-volume namespaces: Isolate logs from specific high-traffic applications

Compliance: Collect logs only from namespaces with compliance requirements

Multiple Namespace Pipelines​

Run multiple Expanso pipelines to collect from different namespaces:

production-logs.yaml:

input:
  subprocess:
    name: kubectl
    args: [logs, --namespace=production, --selector=app=web-app, --follow]
output:
  aws_s3:
    bucket: production-logs

staging-logs.yaml:

input:
  subprocess:
    name: kubectl
    args: [logs, --namespace=staging, --selector=app=web-app, --follow]
output:
  aws_s3:
    bucket: staging-logs

Wrap each fragment in its own job specification before deploying:

expanso-cli job deploy production-logs.yaml
expanso-cli job deploy staging-logs.yaml

Namespace Patterns​

Collect from multiple specific namespaces: Run separate pipelines for each

Exclude system namespaces: Run only the namespace-specific pipelines you need; kubectl logs does not support --all-namespaces

Dynamic namespace selection: Use environment variables:

args:
  - logs
  - --namespace=${NAMESPACE}
  - --selector=app=web-app
  - --follow

Next Steps​