Skip to main content

OpenShift Single-Node at the Edge

Monitor and manage Single-Node OpenShift (SNO) deployments at edge locations with Expanso. Deploy Expanso directly on the OpenShift node to collect logs, monitor cluster health, and forward operational data to your existing storage and monitoring systems. Cloud enrollment, destination services, credentials, and the OpenShift CLI are separate dependencies.

What is Single-Node OpenShift?​

Single-Node OpenShift (SNO) is Red Hat's solution for running OpenShift in constrained edge environments where both control plane and worker capabilities run on a single physical or virtual machine.

Ideal for edge scenarios:

  • Confined physical spaces (retail stores, factories, remote sites)
  • Intermittent network connectivity to central data centers
  • Resource-constrained environments
  • Locations requiring zero-touch operations

Plan capacity for your OpenShift release: consult the SNO system requirements. Reserve additional CPU, memory, and storage for your applications, collectors, and outage buffers; sizing depends on workload and retention requirements.

Expanso Resource Usage

The resource requests and limits below are example starting values, not measured minimum requirements. Profile representative log volume, pipeline processing, and retry behavior before choosing production limits.

Why Use Expanso with Single-Node OpenShift?​

Challenge: SNO deployments at edge locations need monitoring and log collection, but network connectivity may be intermittent.

Solution: Deploy Expanso on the SNO node itself to collect logs and metrics locally, then batch and send to central storage when connectivity is available.

Benefits:

  • Adjustable resource allocation: Set requests and limits for the workload
  • Outage buffering: Use durable storage and monitor available capacity
  • Automatic batching: Optimizes network usage
  • Local processing: Transform records on the SNO node before forwarding
  • Local deployment: Runs directly on the OpenShift node

Deploy Expanso on Single-Node OpenShift​

Deploy the Expanso Edge agent as a DaemonSet on your SNO cluster, where it enrolls with your workspace using a bootstrap token. The full manifest, the bootstrap and kubeconfig secrets, and the custom-image requirement for oc-based log collection are covered in the deployment guide:

Deploy Expanso on Single-Node OpenShift →

Once the agent is enrolled, deploy the pipeline configurations below as jobs from Expanso Cloud. Each pipeline block is a fragment: place it under config in a job with name and type: pipeline, as shown in the quickstart. Configure oc, kubeconfig, destination credentials, and NODE_NAME, CLUSTER_NAME, and LOCATION in the edge environment.

Collect OpenShift Logs​

Stream logs from pods matching app=point-of-sale in production to S3. Replace these values with your workload. oc logs follows the matching pods available when it starts; it does not continuously discover new pods. Use a Kubernetes log collector when you need collection across pod replacement. Restarts can replay records, so account for duplicates. The --prefix output identifies pod and container, not namespace. Keep the namespace in the command, document field, and metadata aligned. The archive preserves the first record's metadata; the S3 path reads that metadata because archived JSONL is not one JSON object. Each object key includes a UUID to prevent later batches from overwriting earlier ones.

input:
  subprocess:
    name: oc
    args:
      - logs
      - --namespace=production
      - --selector=app=point-of-sale
      - --all-containers=true
      - --prefix=true
      - --follow
      - --max-log-requests=10
      - --since=10m
    codec: lines
    restart_on_exit: true
pipeline:
  processors:
    - mapping: |
        root.raw_log = content().string()
        root.timestamp = now()
        root.namespace = "production"
        meta namespace = "production"
        let parts = content().string().re_find_all_submatch("^\\[pod/([^/]+)/([^\\]]+)\\] (.*)$")
        root.pod = $parts.index(0).index(1)
        root.container = $parts.index(0).index(2)
        root.message = $parts.index(0).index(3)
        root.node_name = env("NODE_NAME")
        root.cluster_name = env("CLUSTER_NAME")
        root.location = env("LOCATION")
        root.deployment_type = "single-node-openshift"
output:
  aws_s3:
    bucket: edge-openshift-logs
    path: sno/${! env("CLUSTER_NAME") }/${! now().ts_format("2006-01-02") }/${! metadata("namespace") }/${! uuid_v4() }.jsonl
    batching:
      count: 1000
      period: 5m
      processors:
        - archive:
            format: lines

What this does:

  • Follows all containers in the selected pods
  • Parses pod and container metadata and attaches the configured namespace
  • Adds SNO-specific context (node, cluster, location)
  • Batches logs to minimize network usage
  • Writes to S3 organized by cluster and date

Monitor Cluster Health​

Check SNO cluster health and send metrics to central monitoring. Branch processors preserve each command result in the same report. CrashLoopBackOff is a container waiting reason, not a pod phase. The report below covers selected signals; Pending pods, application readiness, and site-specific conditions may need additional checks. Monitor command and mapping errors so missing permissions or API failures do not become healthy reports:

input:
  generate:
    interval: 60s
    mapping: |
      root.check_time = now()
      root.cluster = env("CLUSTER_NAME")
pipeline:
  processors:
    - branch:
        request_map: root = ""
        processors:
          - command:
              name: oc
              args_mapping: '["get", "nodes", "-o", "json"]'
        result_map: root.nodes = content().parse_json().items
    - branch:
        request_map: root = ""
        processors:
          - command:
              name: oc
              args_mapping: '["get", "clusteroperators", "-o", "json"]'
        result_map: root.operators = content().parse_json().items
    - branch:
        request_map: root = ""
        processors:
          - command:
              name: oc
              args_mapping: '["get", "pods", "--all-namespaces", "-o", "json"]'
        result_map: root.pods = content().parse_json().items
    - mapping: |
        root = this
        root.node_ready = this.nodes.length() > 0 && this.nodes.all(n -> n.status.conditions.any(c -> c.type == "Ready" && c.status == "True"))
        root.degraded_operators = this.operators.filter(op -> op.status.conditions.any(c -> c.type == "Degraded" && c.status == "True")).map_each(op -> op.metadata.name)
        root.all_operators_healthy = this.operators.length() > 0 && this.operators.all(op -> op.status.conditions.any(c -> c.type == "Available" && c.status == "True") && !op.status.conditions.any(c -> (c.type == "Degraded" || c.type == "Progressing") && c.status == "True"))
        root.total_pods = this.pods.length()
        root.running_pods = this.pods.filter(p -> p.status.phase == "Running").length()
        root.failed_pods = this.pods.filter(p -> p.status.phase == "Failed" || p.status.containerStatuses.or([]).any(c -> c.state.waiting.reason.or("") == "CrashLoopBackOff")).map_each(p -> {"namespace": p.metadata.namespace, "name": p.metadata.name, "phase": p.status.phase})
    - mapping: |
        root.health_report = {
          "cluster": this.cluster,
          "location": env("LOCATION"),
          "timestamp": this.check_time,
          "node_ready": this.node_ready,
          "operators_healthy": this.all_operators_healthy,
          "degraded_operators": this.degraded_operators,
          "total_pods": this.total_pods,
          "running_pods": this.running_pods,
          "failed_pods": this.failed_pods,
          "cluster_healthy": this.node_ready && this.all_operators_healthy && this.failed_pods.length() == 0
        }
output:
  switch:
    cases:
      - check: '!this.health_report.cluster_healthy'
        output:
          broker:
            pattern: fan_out
            outputs:
              - http_client:
                  url: https://alerts.company.com/sno-health
                  verb: POST
                  headers:
                    Content-Type: application/json
              - aws_s3:
                  bucket: sno-health-alerts
                  path: alerts/${! env("CLUSTER_NAME") }/${! timestamp_unix() }-${! uuid_v4() }.json
      - output:
          http_client:
            url: https://metrics.company.com/sno-health
            verb: POST
            batching:
              count: 10
              period: 5m

Monitor Resource Usage​

Track CPU and memory on the SNO node. oc adm top requires a functioning metrics API; it does not report disk usage. This parser expects the standard single-node tabular output and preserves memory quantities such as Mi rather than labeling them raw bytes. Namespace totals below count pods, not resource consumption. Configure separate storage monitoring:

input:
  generate:
    interval: 60s
    mapping: root = {}
pipeline:
  processors:
    - branch:
        request_map: root = ""
        processors:
          - command:
              name: oc
              args_mapping: '["adm", "top", "node", "--no-headers"]'
        result_map: root.node_usage = content().string()
    - branch:
        request_map: root = ""
        processors:
          - command:
              name: oc
              args_mapping: '["adm", "top", "pods", "--all-namespaces", "--no-headers"]'
        result_map: root.pod_usage = content().string()
    - mapping: |
        let parts = this.node_usage.trim().re_replace_all("\\s+", " ").split(" ")
        root.node_name = $parts.index(0)
        root.cpu_cores = $parts.index(1)
        root.cpu_percent = $parts.index(2).trim("%").number()
        root.memory_quantity = $parts.index(3)
        root.memory_percent = $parts.index(4).trim("%").number()
        root.cluster = env("CLUSTER_NAME")
        root.timestamp = now()
        root.pod_metrics = this.pod_usage.trim().split("\n").filter(l -> l != "").map_each(line -> line.trim().re_replace_all("\\s+", " ").split(" ")).map_each(cols -> {"namespace": cols.index(0), "pod": cols.index(1), "cpu": cols.index(2), "memory": cols.index(3)})
    - mapping: |
        root = this
        root.namespace_pod_counts = this.pod_metrics.fold({}, item -> item.tally.assign({item.value.namespace: item.tally.get(item.value.namespace).or(0) + 1}))
        root.resource_alert = {"high_cpu": this.cpu_percent > 80, "high_memory": this.memory_percent > 85, "cluster": this.cluster, "timestamp": this.timestamp}
output:
  broker:
    pattern: fan_out
    outputs:
      - http_client:
          url: https://metrics.company.com/sno-resources
          verb: POST
          batching:
            count: 20
            period: 5m
      - switch:
          cases:
            - check: this.resource_alert.high_cpu || this.resource_alert.high_memory
              output:
                http_client:
                  url: https://alerts.company.com/sno-resources
                  verb: POST

Collect Specific Application Logs​

Focus on logs from specific namespaces (e.g., production apps):

input:
  subprocess:
    name: oc
    args:
      - logs
      - --namespace=production
      - --all-containers=true
      - --follow
      - --selector=app=point-of-sale
      - --max-log-requests=10
    codec: lines
    restart_on_exit: true
pipeline:
  processors:
    - mapping: |
        # Parse and structure logs
        root = content().string().parse_json().catch({
          "message": content().string(),
          "level": "info"
        })
        root.cluster = env("CLUSTER_NAME")
        root.location = env("LOCATION")
        root.app = "point-of-sale"
        root.timestamp = now()
output:
  broker:
    pattern: fan_out
    outputs:
      - opensearch:
          urls:
            - https://opensearch.company.com:9200
          index: sno-pos-logs-${! now().ts_format("2006-01-02") }
          action: index
          id: ${! uuid_v4() }
          batching:
            count: 100
            period: 10s
      - aws_s3:
          bucket: sno-app-logs
          path: pos/${! env("CLUSTER_NAME") }/${! timestamp_unix() }-${! uuid_v4() }.jsonl
          batching:
            count: 5000
            period: 10m
            processors:
              - archive:
                  format: lines

Offline-Resilient Configuration​

Handle intermittent connectivity with a SQLite buffer on a persistent volume and retries. Replace the deployment guide's example emptyDir with storage that survives pod replacement before relying on this path for retention. Give the edge service write permission, size the volume for the expected outage, and alert before it fills:

input:
  subprocess:
    name: oc
    args:
      - logs
      - --namespace=production
      - --selector=app=point-of-sale
      - --all-containers=true
      - --follow
      - --max-log-requests=10
    codec: lines
    restart_on_exit: true
pipeline:
  processors:
    - mapping: |
        root.message = content().string()
        root.cluster = env("CLUSTER_NAME")
        root.timestamp = now()
buffer:
  sqlite:
    path: /var/lib/expanso/sno-logs.sqlite
output:
  retry:
    max_retries: 0
    backoff:
      initial_interval: 30s
      max_interval: 10m
      max_elapsed_time: 0s
    output:
      aws_s3:
        bucket: sno-logs
        path: logs/${! env("CLUSTER_NAME") }/${! timestamp_unix() }-${! uuid_v4() }.jsonl
        batching:
          count: 5000
          period: 5m
          processors:
            - archive:
                format: lines

What this does:

  • Stores accepted records in SQLite; disk capacity determines retention
  • Retries S3 output without a configured retry-count or elapsed-time limit
  • Uses exponential backoff from 30 seconds up to 10 minutes
  • Resumes delivery when the destination recovers; recovery rate depends on throughput
  • Does not protect against disk loss, log rotation before collection, or duplicate records after a command restart

Service Account Setup​

Create RBAC for Expanso to read the cluster-wide health and metrics used above. For namespace-only log collection, prefer a narrower Role and RoleBinding. Create expanso-system before applying these resources.

Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: expanso-edge
  namespace: expanso-system

ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: expanso-edge-reader
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "nodes", "namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["nodes", "pods"]
  verbs: ["get", "list"]

ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: expanso-edge-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: expanso-edge-reader
subjects:
- kind: ServiceAccount
  name: expanso-edge
  namespace: expanso-system

Apply all three with:

oc apply -f expanso-serviceaccount.yaml
oc apply -f expanso-clusterrole.yaml
oc apply -f expanso-clusterrolebinding.yaml

Best Practices for SNO​

1. Resource Allocation​

# Example starting values; measure and tune for your workload
resources:
  requests:
    cpu: 100m      # 0.1 CPU cores
    memory: 128Mi  # 128 MB RAM
  limits:
    cpu: 500m      # 0.5 CPU cores max
    memory: 512Mi  # 512 MB RAM max

These values are not a performance guarantee. Low CPU limits can throttle processing; a memory limit can terminate a busy collector. Measure representative load and reserve capacity for the OpenShift control plane and applications.

2. Use Batch Processing​

output:
  aws_s3:
    batching:
      count: 5000       # Larger batches for SNO
      period: 10m       # Longer periods to reduce network

Reduces network overhead critical for edge deployments.

3. Filter Logs Early​

pipeline:
  processors:
    # Only send WARN and ERROR logs
    - mapping: |
        root = if ["warn", "warning", "error", "fatal"].contains(this.level.or("").lowercase()) { this } else { deleted() }

This expects a parsed level field and explicitly discards other records. Parse plain-text severity first if those records must be retained.

4. Add Location Context​

processors:
  - mapping: |
      root.cluster_name = env("CLUSTER_NAME")
      root.location = env("LOCATION")
      root.deployment_type = "single-node-openshift"

Essential for multi-site deployments.

Troubleshooting​

oc Command Not Found​

Solution: Use full path or install OpenShift CLI in Expanso container:

# Add to Dockerfile
RUN curl -LO https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz && \
    tar -xzf openshift-client-linux.tar.gz -C /usr/local/bin oc

Permission Denied​

Solution: Verify service account permissions:

oc auth can-i get pods --all-namespaces --as=system:serviceaccount:expanso-system:expanso-edge

High Resource Usage​

Solution: Profile the bottleneck, narrow the selected workload, and tune batching within memory limits. This is a one-off snapshot of the selected pods' recent logs; it is not a continuous or scheduled collector. Re-running it can produce overlapping records:

input:
  subprocess:
    name: oc
    args:
      - logs
      - --all-containers=true
      - --namespace=production
      - --selector=app=point-of-sale
      - --since=5m  # Only last 5 minutes instead of all logs
    codec: lines

output:
  aws_s3:
    bucket: edge-openshift-logs
    path: 'sno/${! env("CLUSTER_NAME") }/${! timestamp_unix() }-${! uuid_v4() }.log'
    batching:
      count: 10000  # Larger batches
      period: 15m   # Less frequent writes
      processors:
        - archive:
            format: lines

Integration with OpenShift Logging​

Expanso can complement OpenShift's built-in logging. This command reads the logging operator's own pod logs only; it is not a feed of every application log. Confirm the namespace and deployment name for your installed logging version:

# Collect from OpenShift logging stack
input:
  subprocess:
    name: oc
    args:
      - logs
      - --namespace=openshift-logging
      - deployment/cluster-logging-operator
      - --follow
    codec: lines
    restart_on_exit: true

Or forward to external systems that OpenShift logging doesn't support.

Next Steps​

Additional Resources​