# Operations Reference

> Auto-generated from docs.expanso.io documentation

## Operations

# Operations

Run Expanso Edge in production—deployment, configuration, monitoring, and troubleshooting.

## Expanso Cloud

# Expanso Cloud

Expanso Cloud gives teams one place to build pipelines, choose where they run,
and follow their operation across a fleet of edge nodes. Use the visual editor
or YAML, test transformations before deployment, and roll out changes with
health checks and rollback controls.

Production pipelines execute on Expanso Edge nodes. You choose the hosts,
source systems, and destinations. Cloud manages configuration and orchestration
and provides a shared view of pipelines, nodes, executions, and telemetry.

## Start with your task

| Task | Guide |
| --- | --- |
| Set up your team and separate environments | [Organizations and workspaces](/operations/cloud/workspaces) |
| Connect nodes and choose where pipelines run | [Fleet management](/operations/cloud/fleet-management) |
| Build, reuse, and test pipeline logic | [Pipeline authoring and testing](/operations/cloud/pipeline-authoring) |
| Release a change gradually and recover a previous version | [Deployments and rollback](/operations/cloud/deployments) |
| Inspect throughput, errors, executions, and logs | [Monitoring and troubleshooting](/operations/cloud/monitoring) |
| Configure team access and automation credentials | [Access and API keys](/operations/cloud/access-and-api-keys) |
| Reference credentials from secret providers | [Secrets and configuration](/operations/cloud/secrets) |
| Get help building and fixing a pipeline | [Pipeline assistant](/operations/cloud/pipeline-assistant) |
| Automate deployments and operate repeatable environments | [CLI and delivery workflows](/operations/cloud/automation) |
| Understand workspace utilization | [Usage and availability](/operations/cloud/usage-and-availability) |

For a first deployment, follow the [Cloud quickstart](/getting-started/quickstart/).
For a terminal-based workflow, follow [Deploy to Cloud](/getting-started/deploy-to-cloud/).

## How the pieces fit

An **organization** brings your team together. A **workspace** contains the nodes
and pipelines for a particular environment or purpose. A **pipeline** describes
what to read, how to transform it, and where to send it. An **execution** is a
particular instance running on a node. A **deployment** tracks a pipeline
release across its target nodes.

This separation helps operators answer distinct questions: what configuration
should run, where it should run, whether the rollout is progressing, and whether
each execution is healthy.

## Data and configuration

The normal production record path runs from sources through your edge nodes to
your configured destinations. Cloud also handles the information needed to
manage those pipelines, including configuration, status, metrics, and requested
logs. Testing and assistant workflows can send samples or configuration you
provide to Cloud. Review [the authoring workflow](/operations/cloud/pipeline-authoring#choose-test-data)
before using production records in a test or assistant conversation.

## Available controls

The guides describe customer-facing controls. Some panels, including the secrets
editor and usage reporting, appear only when enabled for your account. Pipeline
rollout choices in the console are **Rolling** and **Immediate**. Follow the
availability notes on each page and contact [support@expanso.io](mailto:support@expanso.io)
when you need a control that is not visible.

[LLM text version of the operations documentation](https://docs.expanso.io/llms/operations.txt)

## Team Access and API Keys in Expanso Cloud

# Team Access and API Keys

Expanso Cloud supports shared organization access for people and API keys for
automation. Use organization roles to manage team responsibilities and key
scope to limit which workspaces an integration can access.

## Sign in

Cloud supports passwordless email sign-in and configured social sign-in
providers. Choose one of the methods displayed on the login page. Email magic
links expire; request another if the link is no longer valid.

After sign-in, select the organization and workspace you intend to operate.

## Invite and manage members

In a shared organization's Settings, authorized users can invite colleagues,
inspect pending invitations, change member roles, and remove members. Personal
organizations do not support invitations.

The customer organization roles are **owner**, **admin**, and **member**:

| Permission | Owner | Admin | Member |
| --- | --- | --- | --- |
| Manage workspaces, pipelines, and templates | Yes | Yes | Yes |
| Manage bootstrap tokens | Yes | Yes | Yes |
| Create and view API-key metadata | Yes | Yes | Yes |
| Revoke own API keys | Yes | Yes | Yes |
| Revoke another member's API keys | Yes | Yes | No |
| Administer members and invitations | Yes | Yes | No |
| Update organization settings | Yes | Yes | No |
| Delete the organization | Yes | No | No |

Roles apply at the organization level. **Member is not a read-only role.**
Organization membership should reflect who is trusted to operate its resources.
The member-management UI restricts changes to owner and self roles.

## Create an automation key

Open the workspace's **Keys** view or **API Access** dialog to create a key
scoped to that workspace. Organization Settings also provides key creation,
where you can choose one workspace or all workspaces in the organization.
Give the key a name that identifies the integration and choose an expiry.
A no-expiry option is available when appropriate.

Copy the raw key when it is shown and store it securely. It is shown once;
subsequent key listings expose metadata rather than the raw credential.
Cloud rejects expired or revoked keys and records last use.

Prefer a workspace-specific key when an integration only needs that workspace.
Scope limits the workspaces a key can access; it should not be interpreted as a
custom read-only role or a separate permission policy for each pipeline.

## Connect the CLI

The workspace overview's **API Access** dialog provides the actual endpoint and
CLI profile instructions. Use those instructions to configure automation, then
verify that the profile reaches the intended workspace.

Follow [Deploy to Cloud](/getting-started/deploy-to-cloud/) for the full setup.
Keep keys out of checked-in YAML and shell history where possible, and use your
CI system's secret handling for automated deployments.

## API keys and bootstrap tokens

| Credential | Purpose | Scope |
| --- | --- | --- |
| API key | CLI and API operations | Selected workspace or organization workspaces, according to key scope. |
| Bootstrap token | Register an edge node | The workspace that issued the token. |

Revoking an enrollment token and revoking an API key are different operations.
Use the credential's own management view and check its purpose before changing it.
For node onboarding, see [Fleet management](/operations/cloud/fleet-management).

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Automate Expanso Cloud with the CLI

# Automate Expanso Cloud with the CLI

Use the Cloud console to design and inspect pipelines, then use the workspace
API and `expanso-cli` to make deployment steps repeatable. A job definition can
carry pipeline logic and deployment settings in a file that your team reviews
and retains in version control.

## Connect to the right workspace

Open **API Access** from the workspace overview to obtain the endpoint and
profile instructions. Create an API key under **Keys**. Use the generated
instructions for your workspace rather than substituting the Cloud console URL
as an API endpoint.

A bootstrap token registers nodes; an API key authenticates automation. See
[Access and API keys](/operations/cloud/access-and-api-keys) for their different scopes and
lifecycles. Keep credentials in your automation system's secret store rather
than in the job file or repository.

The [Deploy to Cloud guide](/getting-started/deploy-to-cloud/) walks through
profile setup, node verification, job validation, and deployment. The
[CLI reference](/references/cli/expanso-cli/) and
[workspace API reference](/references/api/) document individual operations.

## Build a repeatable release

1. Retain the complete job definition, including node selection and rollout
   settings, alongside the pipeline configuration.
2. Test transformations with representative, approved samples. Validate the
   pipeline and job formats with the tools described in
   [Testing and debugging](/getting-started/testing-debugging/).
3. Review the change in your existing source-control workflow.
4. Select the intended workspace profile and deploy the job.
5. Check deployment progress and execution health. Successful submission means
   the job was accepted; it does not prove that a matching node executed it.
6. Compare versions and use the documented rollout or rollback controls when
   you need to change course.

Cloud's [deployment view](/operations/cloud/deployments) gives operators the corresponding
progress and control surface for a release initiated through automation.

## Organize staging and production

Separate workspaces can represent development, staging, and production. Reuse
pipeline logic while making each environment's source endpoints, destination
endpoints, secret references, labels, and deployment policy explicit.

Treat this as a delivery workflow owned by your team. Copying or deploying a
job into another workspace does not automatically create external credentials,
source data, destination tables, or host resources there. Confirm that the
selected nodes can resolve every referenced dependency before releasing.

## Plan data migration separately

Rolling deployment controls how a new pipeline version reaches nodes. Historical
backfill also requires decisions about the source range, destination writes,
deduplication, and completion criteria. Configure those through the selected
components and verify the resulting data at the destination.

Deployment progress reports release progress across nodes. It is not a count of
historical records copied or reconciled. Likewise, restoring a previous pipeline
version restores its configuration; it does not undo records already written.

For a worked data movement example, see [Data migration](/recipes/data-migration/).
For retry and delivery behavior, consult the specific input, buffer, and output
[component references](/components/).

[LLM text version of the operations documentation](https://docs.expanso.io/llms/operations.txt)

## Pipeline Deployments, Rollouts, and Rollback in Expanso Cloud

# Pipeline Deployments, Rollouts, and Rollback

Expanso Cloud provides release controls for pipelines running across distributed
nodes. Choose how quickly a new version reaches the fleet, monitor its progress,
and use pause, resume, or rollback when a release needs attention.

## Choose a rollout strategy

The Cloud console exposes two strategies:

| Strategy | Behavior | When to use it |
| --- | --- | --- |
| **Rolling** | Deploy in waves, with health evaluation during the rollout. | Limit how much of the fleet receives a change at once. |
| **Immediate** | Deploy to all target nodes at once. | Apply a change broadly when a staged release is unnecessary. |

For Rolling deployments, **Wave Size (%)** sets the percentage used for
parallel updates. The form also exposes **Max Failed (%)** and
**Auto-rollback on failure**. Choose these settings to match the size of the
fleet and the consequences of a failed update.

The current console does not offer a selectable canary strategy. A small,
explicitly selected node group can still be useful for a test deployment, but
that does not create an automatic traffic-splitting or canary-promotion workflow.

## Configure a deployment

1. Build and validate the pipeline configuration.
2. Set its name, description, and labels.
3. Review the [node selector](/operations/cloud/fleet-management#select-deployment-targets).
4. Set runtime behavior, including restart policy, priority, and timeouts.
5. Choose Rolling or Immediate and review the applicable failure controls.
6. For Rolling, configure health evaluation and parallelism.
7. Submit the deployment and open its progress view.

Runtime restart choices are on failure, always, or never. Execution, queue, and
total timeouts let you bound different stages of a job's lifecycle; the form
supports zero to mean no timeout. These are execution controls, not a guarantee
that a restarted pipeline resumes from a particular source offset.

## Configure rolling health checks

Rolling deployment settings include:

| Setting | What it controls |
| --- | --- |
| Deadline | Maximum wait for an execution to become healthy. |
| Interval | Duration of each health evaluation window. |
| Maximum error rate | Error-rate threshold used in health evaluation. |
| Success threshold | Consecutive healthy windows needed to establish health. |
| Failure threshold | Consecutive unhealthy windows needed to establish failure. |

Review these together with the rollout's failure budget and automatic-rollback
setting. One transient error is not the same as a rollout meeting its configured
failure condition. Custom health-check settings in the creation form apply to
Rolling deployments; the form does not submit them for Immediate deployment.

## Follow release progress

Open a pipeline's deployment history and select a version. The deployment detail
shows updated, pending, and failed node counts, rollout percentage, current and
total waves, strategy, trigger, and timestamps. Its execution list is filtered
to that version, and its timeline shows deployment events.

Use these views to distinguish a pipeline that was accepted by Cloud from one
that has reached its target nodes. If progress stalls, check node connectivity,
selectors, execution status, and the configured health and failure settings.

## Pause and resume

While a deployment is running, use **Pause** to stop further rollout progress.
When the rollout is paused, use **Resume** to continue. Pausing a rollout is a
release-control action; it is not a command to stop every pipeline execution.

Inspect the updated nodes and their telemetry before resuming a release that
you paused for investigation.

## Compare versions and roll back

Cloud retains pipeline versions and lets you inspect their specifications.
The deployment detail's **Diff** view compares the selected version with its
predecessor. This gives reviewers a concrete view of what changed.

When a valid rollback target is available, **Rollback** presents the target
version for confirmation. A rollback creates a new deployment using the previous
configuration. It does not erase deployment history or undo records already
written to an external destination.

Automatic rollback is separately configurable during deployment. Its outcome
depends on the rollout settings and an available rollback target; an initial
deployment has no earlier configuration to restore.

## Stop or reassess a pipeline

The pipeline list supports **Stop** and **Start** or **Restart**, including
bulk operations such as **Restart Pipelines**. Start and Restart use the API's rerun operation. Stopping
controls the job's lifecycle. Rerun asks the scheduler to reassess the job; it should not be treated as a promise to forcibly restart every healthy
execution or replay a chosen historical range.

## Understand the recovery boundary

Configuration rollback, runtime recovery, and data reconciliation solve different
problems. Source offsets, checkpoints, durable buffers, and duplicate handling
depend on the selected components and pipeline design. Consult the relevant
[component references](/components/) and
[restart guidance](/operations/troubleshooting/pipeline-progress-lost-after-restart/)
before relying on a release workflow to preserve processing state.

For moving definitions between workspaces or planning a backfill, see
[CLI and delivery workflows](/operations/cloud/automation).

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Connect and Manage an Expanso Cloud Fleet

# Connect and Manage Your Fleet

Expanso Cloud connects pipeline management to the hosts where your data lives.
Register an Expanso Edge node with a workspace, confirm that it is connected,
and select which pipelines it should run.

## Add a node

Open **Nodes → Add Node** in the intended workspace. Cloud offers instructions to:

- Install and bootstrap Expanso Edge.
- Bootstrap an existing installation.
- Run Expanso Edge in Docker.

Choose a token validity period and copy the instructions appropriate to your
host. For a non-Docker installation, follow the provided instructions to start
Expanso Edge after bootstrap. A successful installation alone does not prove
that the node has connected to Cloud.

Return to the node list and verify the connection state. The
[node quickstart](/getting-started/quickstart/deploy-node/) provides the full
onboarding walkthrough.

## Manage bootstrap tokens

Bootstrap tokens enroll nodes into a particular workspace. Under **Keys**, you
can create tokens with a description and expiry, inspect their creation and
last-use information, and revoke them. Non-expiring tokens are available when
that fits your enrollment process; time-limited tokens are useful for bounded
installation work.

Store tokens securely. An API key and a bootstrap token have different purposes:
use an [API key](/operations/cloud/access-and-api-keys) for CLI or API operations.

## Inspect fleet health

The fleet view summarizes connected, disconnected, and lost nodes and the
connected share of the fleet. Filter the node table by connection state or
operating system, refresh it, and choose the columns useful to your team.

Open a node to inspect its hostname, operating system, architecture, agent
version, last heartbeat, connection timestamps, status message, labels, and
executions. These details help distinguish an unavailable host from a pipeline
problem on an otherwise connected node.

The node Monitoring view provides CPU utilization and memory usage over a
selected time range. For connection timing, see
[workspace settings](/operations/cloud/workspaces#tune-connection-health).

## Select deployment targets

The pipeline deployment form supports:

| Selector | Use |
| --- | --- |
| Matching labels | Select nodes with the required key/value properties; all label constraints must match. |
| Expressions | Select by conditions such as membership, absence, or inequality. |
| Explicit node IDs | Choose particular nodes for a deployment. |

With no selector constraints, the pipeline targets all eligible nodes in the
workspace. Review this choice before deploying a job intended for a single
site or test host.

Labels and selectors let the same pipeline logic serve different host groups.
They describe placement; they do not copy host files, provision external
services, or change source permissions.

## Remove a node record

The node table supports selected-node deletion with confirmation. Review which
records are selected before confirming. Treat node-record removal separately
from uninstalling software or shutting down a host; do not assume the console
has performed those host operations.

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Monitor and Troubleshoot Expanso Cloud Pipelines

# Monitoring and Troubleshooting

Expanso Cloud brings fleet, pipeline, component, and execution information into
the same workspace. Start with the scope of the problem, then narrow the view to
the affected version, node, or component.

## Choose the right view

| View | What you can inspect |
| --- | --- |
| Nodes | Connected, disconnected, and lost nodes, host details, and agent versions. |
| Node Monitoring | CPU utilization and memory usage over a selected time range. |
| Pipeline Monitoring | Input bytes, output bytes, and orchestration errors, with deployment context. |
| Selected pipeline component | Component-appropriate byte metrics and processor latency percentiles. |
| Deployment detail | Release progress, version-specific executions, and event timeline. |
| Pipeline Logs | Live logs from a selected connected node. |

Use the time-range and refresh controls to investigate a period of interest.
Metrics arrive on an export interval, so a recent operational change may appear
in the charts after a delay.

Processor latency describes processing at that component. Do not interpret it
as source-data freshness or end-to-end delivery latency unless your pipeline
measures those quantities explicitly.

## Follow a failure from rollout to execution

1. Open the pipeline and check its status and deployment context.
2. If the issue began with a release, inspect that deployment's progress and
   compare the configuration with the previous version.
3. Open the affected execution and identify its node and reported state.
4. Check the node's connection state, recent heartbeat, CPU, and memory.
5. Inspect component traffic and, for processors, latency to narrow the affected processing step.
6. Tail logs from the connected node to investigate the current error.

This workflow connects a release to its running instances. A healthy control-plane
response or accepted job submission alone does not prove successful output.
Verify delivery at the destination when that is the outcome you need.

## Stream live logs

Open the pipeline's **Logs** view, select a node, and start streaming. Use
severity and text filters to focus the display. You can stop streaming, clear
the displayed entries, and control auto-scroll.

Live streaming requires a connected node and supported versions of both the
workspace runtime and Expanso Edge. The current minimum for this feature is
**v2.1.18** on both sides. Unknown or unsupported versions disable the stream;
contact support if you need help checking your workspace's version.

The browser keeps up to 10,000 streamed entries. This is a live troubleshooting
view, not a promise of historical log retention or a durable archive. Logs may
contain payload values or sensitive diagnostics, so configure pipeline logging
with that in mind.

## Integrate with your observability tools

Use the [OpenTelemetry guide](/operations/monitoring/opentelemetry/) for documented
telemetry export patterns. The Cloud console's built-in charts and an external
observability platform have different controls; configure alert rules and
retention in the system responsible for them.

For pipeline recovery behavior, use the specific
[troubleshooting guides](/operations/troubleshooting/) and the relevant component
references. Cloud visibility helps diagnose a failure; delivery and checkpoint
behavior still depend on the pipeline's inputs, processing, buffering, and outputs.

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Use the Expanso Cloud Pipeline Assistant

# Pipeline Assistant

The Cloud pipeline assistant helps you create, explain, and repair pipeline
configuration. It works with the current YAML and your conversation, and can
retrieve component, transformation-language, and example documentation.

Use it when you need help selecting a component, writing a transformation, or
understanding a configuration error. You retain the normal editor and deployment
workflow for reviewing and releasing the result.

## Work with the current pipeline

1. Open the pipeline editor and its assistant panel.
2. Describe the source, transformation, and destination you need, or explain the
   error you are investigating.
3. Review the proposed configuration and the assistant's validation feedback.
4. Use **Apply** to place the proposed configuration into the editor.
5. Inspect and test the result, then save or deploy it through the normal controls.

The assistant receives the current pipeline YAML as context. It can help refine
an existing definition rather than requiring you to describe the whole pipeline
again. Applying a suggestion changes the editor; it does not itself deploy to
production.

## Understand validation feedback

The assistant has a pipeline-validation tool that checks syntax, component
types, field values, and required fields. If the validation service is unavailable,
feedback can be limited to syntax checks. Treat that narrower result as such and
run full validation when it is available.

Validation of a configuration does not prove source connectivity, authorization,
destination writes, performance, or recovery behavior on your nodes. Test those
properties in the intended environment.

## Provide useful context

Include the shape of an example record, the desired output, and any constraints
on the transformation. Use synthetic or redacted samples when records contain
sensitive information. Avoid putting passwords, tokens, or raw secrets into the
conversation or pipeline YAML.

Cloud receives the conversation and configuration used by the assistant.
This is separate from the normal production record path through your edge nodes.

## Continue through the standard release workflow

Assistant-generated YAML remains editable and can use the same validation,
sample testing, versioning, deployment, and rollback processes as hand-authored
configuration. See [Pipeline authoring](/operations/cloud/pipeline-authoring) and
[Deployments and rollback](/operations/cloud/deployments).

The assistant depends on service availability and configuration. If it is
unavailable, you can continue using the visual and YAML editors.

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Build and Test Pipelines in Expanso Cloud

# Build and Test Pipelines

Use the Cloud pipeline editor to connect inputs, transformations, and outputs,
then test the transformation logic before choosing deployment targets.
The visual graph and YAML editor provide two ways to work with the pipeline
configuration.

## Start a pipeline

From the workspace's Pipelines view, create a pipeline. Start from scratch,
upload a YAML configuration, or choose a public or organization template.
Templates let teams reuse a configuration and adapt it for a new task.

Use the component picker and configuration panel to add or edit inputs,
processors, and outputs. The [component catalog](/components/) explains each
component's fields and behavior. Use the YAML editor when direct configuration
editing is more convenient.

The current creation wizard expects an input, an output, and at least one
processor before continuing. Consult the
[configuration formats guide](/getting-started/configuration-formats/) when
moving between a pipeline configuration and the full job format used by the CLI.

## Validate before continuing

Cloud lints the configuration and displays issues in the editor. The wizard
validates again before advancing to deployment settings. Correct reported
errors before proceeding.

Configuration validation checks what can be established from the definition.
It does not establish that a production node can reach an endpoint, obtain a
credential, or write to a destination. Check those dependencies in the intended
deployment environment as well.

## Test transformations

Provide sample input to test transformation behavior while editing. Cloud
returns transformed output together with errors and warnings. Adjust the
configuration and sample to investigate parsing, field mapping, filtering, and
other processing logic.

Use examples that exercise the cases you expect in production: normal records,
missing fields, malformed values, and records that should be rejected or routed
differently. The test excludes inputs and outputs, can skip unsupported processors, and
returns at most 100 output messages. Review warnings to see which processing
steps were exercised. A successful sample test helps you review the logic; it
is not a throughput benchmark or an end-to-end connector test.

### Choose test data

Transformation tests run in a browser worker when available and can fall back
to Cloud's testing service. Use synthetic or redacted records when the original
data should remain on your infrastructure. The same care applies to configuration and
samples pasted into the [pipeline assistant](/operations/cloud/pipeline-assistant).

Production execution follows a different path: the deployed pipeline runs on
your selected edge nodes and sends records to its configured destinations.

## Reuse pipeline configuration

YAML files and templates make pipeline logic reusable. Before deploying a reused
configuration, review source and destination addresses, secret references, node
selectors, and rollout settings for the new environment. A template does not
provision its external dependencies.

### Maintain organization templates

Use the pipeline catalog to create an organization template with a name,
description, tags, and YAML content. Edit an existing template as your team's
preferred configuration evolves, or download its configuration for review and
reuse outside the catalog. Public templates provide additional starting points.

A template is a reusable starting configuration. Editing it is not the same as
releasing an update to pipelines previously created from it; review and deploy
those pipelines through their normal lifecycle.

When the configuration is ready, continue to
[Deployments and rollback](/operations/cloud/deployments). For local validation and additional
debugging tools, see [Testing and debugging](/getting-started/testing-debugging/).

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Secrets and Pipeline Configuration in Expanso Cloud

# Secrets and Pipeline Configuration

Secret references let a pipeline describe which credential it needs without
repeating the credential value in each component. Configure a provider, define
a named reference, and use that reference in the component configuration.

The **Secrets** editor is available when enabled for your account. If the control
is not visible, contact [support@expanso.io](mailto:support@expanso.io).

## Configure a provider

In the pipeline editor, open Secrets and select a provider type. The current
Cloud editor includes:

- File.
- AWS Secrets Manager.
- AWS Systems Manager Parameter Store.
- AWS STS.
- Azure Key Vault.
- HashiCorp Vault.

Supply the provider configuration appropriate to the environment where the
pipeline will execute. Then add named secret mappings and reference them from
component fields. The editor provides views for listing and editing providers
and their secret references.

Provider configuration and references travel with the job specification.
The nodes still need the identity, permissions, network access, and any local
files required to resolve them. A reference is not a copy of the underlying
secret store or a grant of access to it.

## Reuse configuration across environments

Choose stable reference names for credentials used by multiple components.
When reusing pipeline logic in another workspace, verify that its provider
configuration resolves the intended environment's secret rather than a staging
or development value.

Review provider-specific documentation for authentication, refresh, caching,
and rotation behavior. Those details depend on the provider and runtime; the
presence of an editor control alone does not define their behavior.

See [Secret providers](/guides/secrets-management/secret-providers/) and the
[provider reference](/guides/secrets-management/provider-reference/) for the
runtime configuration surface. The runtime reference and the Cloud editor's
provider picker may expose different sets of options.

## Keep test and assistant input appropriate

Avoid putting raw credentials into sample records or assistant conversations.
Configuration submitted to Cloud can include the values you type into it.
Use named references and synthetic or redacted samples when evaluating a
pipeline in the editor.

For access to Cloud itself, use [organization roles and API keys](/operations/cloud/access-and-api-keys).
For credentials used by a source or destination, use the relevant component and
secret-provider settings.

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Workspace Usage and Feature Availability in Expanso Cloud

# Usage and Feature Availability

Cloud provides operational views for your workspaces and, when enabled, a usage
view for understanding connected-node utilization over time.

## Review workspace usage

Open the workspace's **Usage** view when it is available. Select a UTC month to
see the average number of connected nodes and the hourly chart. Use this to
understand how the fleet's connected capacity changes through the month.

The usage view reports connected-node activity. Consult your commercial
agreement for billing terms, or use [Expanso pricing](https://www.expanso.io/pricing) or contact
[support@expanso.io](mailto:support@expanso.io).

## Check feature prerequisites

| Capability | Prerequisite |
| --- | --- |
| Secrets editor | Enabled for your account. |
| Workspace Usage | Enabled for your account. |
| Live pipeline logs | Connected node and supported workspace/Edge versions; currently v2.1.18 or later on both sides. |
| Pipeline assistant | Assistant service configured and available. |
| Organization invitations | Shared organization and a role that can manage invitations. |
| Pipeline execution | Connected, eligible nodes and working source/destination dependencies. |

Account enablement, permissions, and runtime compatibility are distinct. A
missing control does not always mean the same thing as an unavailable node or
an invalid pipeline configuration.

## Get help with an operational issue

Before contacting support, gather the workspace and pipeline identifiers,
relevant deployment version, approximate event time, affected node version,
and the error or state you observed. Share only the configuration and logs
needed to investigate, with credentials and sensitive record values removed.

Use [Monitoring and troubleshooting](/operations/cloud/monitoring) to narrow the issue first.
Check [Expanso service status](https://status.expanso.io/) for reported service
incidents, then contact [support@expanso.io](mailto:support@expanso.io) if needed.

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Organizations and Workspaces in Expanso Cloud

# Organizations and Workspaces

Use an organization to collaborate with a team and workspaces to organize its
pipelines and edge nodes. Separate workspaces can represent environments, sites,
or operational responsibilities while keeping the same Cloud interface.

## Choose an organization

Cloud supports personal and shared organizations. A personal organization is
for your own work. Create a shared organization when you need to invite team
members; shared-organization creation also provisions a default workspace.

Select the organization before creating resources. Its membership and roles
determine who can operate its workspaces. Organization Settings provides member
management and invitations for shared organizations. Personal organizations do
not accept invitations.

See [Access and API keys](/operations/cloud/access-and-api-keys) before assigning roles. A member
is an operator with resource-management permissions, not a read-only viewer.

## Create and select a workspace

1. Open the workspace list in the intended organization.
2. Create a workspace and give it a name that explains its purpose.
3. Choose the node-connectivity settings appropriate to the hosts it will manage.
4. Wait for provisioning to complete before connecting nodes or deploying jobs.
5. Open the workspace and use its Nodes, Pipelines, Keys, and Settings views.

Creation is subject to your organization's workspace quota. Contact
[support@expanso.io](mailto:support@expanso.io) if you need additional capacity.

Each workspace provides an API endpoint and its own node-enrollment context.
Keep workspace names and CLI profile names aligned so operators can tell a
staging deployment from a production deployment before submitting a job.

## Tune connection health

Workspace settings expose node-health timing presets and custom values. These
cover the heartbeat interval and the time thresholds used to recognize a node
as connected, disconnected, or lost.

Choose settings that match the network between Cloud and your nodes. A reliable
local connection and an intermittent wide-area connection may need different
tolerances. Short thresholds detect interruptions sooner; longer thresholds
allow more time for a transient connectivity problem to recover.

Review the confirmation when applying timing changes to a running workspace.
These settings determine how Cloud classifies node connection health.

## Plan environment dependencies

Workspace separation gives teams a practical way to organize staging and
production. Before deploying into a workspace, make sure its nodes can reach
the intended sources, destinations, and secret providers. Use labels to identify
host properties and select the right deployment targets.

For a repeatable release process, see [CLI and delivery workflows](/operations/cloud/automation).
For connecting those hosts, see [Fleet management](/operations/cloud/fleet-management).

[LLM text version](https://docs.expanso.io/llms/operations.txt)

## Edge Node Configuration

# Edge Node Configuration

Configuration reference for Expanso Edge nodes.

## Minimal Configuration

Only one field is required to run an edge node:

```yaml
# Core data directory (required)
data_dir: ~/.expanso/edge
```

With just `data_dir` configured:
- The edge node runs in **local mode** (no workspace connection)
- Node name defaults to system hostname
- Subdirectories are created automatically
- API server is enabled on `localhost:9010` (local only, no authentication)
- Logging defaults to `info` level with `console` format

## Common Configurations

### Connect to Your Workspace

To connect to Expanso Cloud (most common):

```yaml
data_dir: ~/.expanso/edge

orchestrator:
  address: nats://orchestrator.expanso.io:4222
```

### Custom Node Name

Set a custom node name instead of using the hostname:

```yaml
data_dir: ~/.expanso/edge
name: edge-production-01
```

### Enable API Server

Enable the local management API:

```yaml
data_dir: ~/.expanso/edge

api:
  listen_addr: localhost:9010
```

**Warning:** The API has no authentication by default. Only enable on trusted networks or add authentication.

### Debug Logging

Enable debug logging for troubleshooting:

```yaml
data_dir: ~/.expanso/edge

log:
  level: debug
  format: text  # Human-readable format
```

## Complete Configuration Reference

All available configuration options:

```yaml
# Node name - defaults to system hostname
name: edge-node-01

# Name provider for automatic name generation
# Options: "cloud", "hostname", "uuid", "machine-id"
name_provider: hostname

# Core data directory (REQUIRED)
data_dir: /var/lib/expanso/edge

# Local mode - run without workspace connection
local_mode: false

# Workspace connection settings
orchestrator:
  address: nats://orchestrator.expanso.io:4222
  require_tls: true
  # Network identifier (optional)
  # Set automatically during bootstrap, or manually for local development
  network_id: ""

# API server configuration
api:
  listen_addr: localhost:9010
  # auth is optional; omit it for an unauthenticated local API.
  # For JWT / identity-provider auth:
  # auth:
  #   organization_id: org_abc123
  #   jwt:
  #     issuer: https://cloud.expanso.io

# Logging configuration
log:
  level: info  # trace, debug, info, warn, error
  format: json # console, json, text

# Telemetry configuration (OpenTelemetry)
telemetry:
  endpoint: localhost:4317
  protocol: grpc  # grpc or http
  export_interval: 30s
  resource_attributes:
    environment: production
    region: us-west-2

# Graceful shutdown timeout
shutdown_timeout: 30s

# Node labels for selection and filtering
labels:
  region: us-west-2
  environment: production
```

## Configuration Sections

### Node Identity

```yaml
# Node name (defaults to hostname)
name: edge-node-01

# Name generation method
name_provider: hostname  # cloud, hostname, uuid, machine-id
```

**Name Providers:**
- `hostname`: Uses system hostname (default)
- `cloud`: Fetches from cloud metadata (AWS, GCP, Azure)
- `uuid`: Generates random UUID
- `machine-id`: Uses system machine ID

### Data Directory

```yaml
# REQUIRED: Core data directory
data_dir: /var/lib/expanso/edge
```

**Subdirectories (created automatically):**
- `state/` - Pipeline state and checkpoints
- `metrics/` - Metrics data
- `temp/` - Temporary files
- `auth/` - Credentials from bootstrap
- `config.d/` - Dynamic configuration

### Workspace Connection

```yaml
orchestrator:
  address: nats://orchestrator.expanso.io:4222
  require_tls: true
  network_id: ""  # Set automatically during bootstrap
```

**Authentication is required.** The edge authenticates to your workspace on every connection; there is no unauthenticated path. It presents credentials obtained during bootstrap (from your bootstrap token) and stored in its data directory, over TLS, and refreshes them automatically. A node cannot connect to your workspace until it has been bootstrapped.

**Key points:**
- Address uses the NATS protocol, secured with TLS (`require_tls`)
- Authentication credentials are obtained during bootstrap and refreshed automatically
- Automatic retry with exponential backoff

**Network ID:**

The `network_id` field identifies which workspace the node belongs to. This is set automatically during bootstrap based on the workspace you're registering the node with. For local development without a workspace, you can set this manually (e.g., `network_id: "local-dev"`).

### Bootstrap TLS Settings

If your self-hosted control plane uses self-signed certificates, you can skip TLS verification during bootstrap.

:::caution[Security Notice]
Only use this when connecting to a control plane you trust. Skipping TLS verification disables certificate validation, which could expose your bootstrap token if connecting to an untrusted server.
:::

**CLI flags:**

| Command | Flag | Description |
|---------|------|-------------|
| `expanso-edge bootstrap` | `--insecure` or `-k` | Skip TLS verification during bootstrap |
| `expanso-edge run` | `--bootstrap-insecure` or `-k` | Skip TLS verification for automatic bootstrap on startup |

**Environment variable:**

```bash
# Skip TLS verification during bootstrap
export EXPANSO_EDGE_BOOTSTRAP_INSECURE=true
```

**Example usage:**

```bash
# Bootstrap with self-signed certificate
expanso-edge bootstrap --token "abc123" --url "https://bootstrap.internal:8443" --insecure

# Or using environment variable
EXPANSO_EDGE_BOOTSTRAP_INSECURE=true expanso-edge bootstrap --token "abc123"
```

:::info[Scope]
This only affects the HTTPS call to the bootstrap service. After bootstrap, the edge node connects to your workspace via NATS with separate TLS settings.
:::

### Custom CA Certificates for NATS

If your orchestrator uses certificates signed by a private CA, the control plane can provide custom CA certificates during bootstrap. These are automatically used for NATS connections.

**How it works:**

1. During bootstrap, the control plane returns custom CA certificates (if configured)
2. The edge node stores these certificates in its connection configuration
3. When connecting to NATS, the edge node trusts both system CAs and the custom CAs

This lets edge nodes connect to orchestrators using your private PKI while still working with public CA-signed certificates.

:::note[Server-Side Configuration]
Custom CA certificates are configured on the control plane, not the edge node. The edge node receives them automatically during bootstrap. See your self-hosted deployment documentation for control plane configuration.
:::

### API Server

```yaml
api:
  listen_addr: localhost:9010
```

**Authentication options:**
- JWT: configure `auth.jwt` for identity provider integration (see below)
- None: omit the `auth` section; the API is then unauthenticated, so keep it bound to localhost

#### JWT Authentication

For browser-based access or identity provider integration (e.g., AWS Cognito):

```yaml
api:
  listen_addr: ":9010"
  auth:
    organization_id: "org_abc123"
    jwt:
      issuer: "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_xxxxx"
```

**Required fields:**
- `jwt.issuer`: JWT issuer URL (e.g., Cognito User Pool URL). JWKS URL is derived automatically by appending `/.well-known/jwks.json`
- `organization_id`: the organization this node belongs to. Tokens must include this organization in their claims

#### CORS Configuration

For browser-based API access, configure allowed origins:

```yaml
api:
  cors:
    allowed_origins:
      - "https://cloud.expanso.io"
      - "https://localhost:*"
```

**Security:**
- Default is `localhost:9010` (local-only)
- Use reverse proxy for HTTPS and remote access

### Logging

```yaml
log:
  level: info
  format: json
```

**Levels:**
- `error` - Only errors
- `warn` - Warnings and errors
- `info` - Informational messages (default)
- `debug` - Detailed debugging
- `trace` - Very detailed tracing

**Formats:**
- `json` - Structured JSON (recommended for production)
- `text` - Human-readable text
- `console` - Colored console output

### Telemetry

```yaml
telemetry:
  endpoint: localhost:4317
  protocol: grpc
  export_interval: 30s
  resource_attributes:
    environment: production
    region: us-west-2
```

**Protocols:**
- `grpc` - OTLP over gRPC (default, port 4317)
- `http` - OTLP over HTTP (port 4318)

#### Metric Filtering

Edge nodes drop high-cardinality metrics by default to reduce telemetry costs. Control this with `drop_metric_prefixes`:

```yaml
telemetry:
  endpoint: localhost:4317
  drop_metric_prefixes:
    - "db."
    - "ncl."
    - "ncltransport."
```

**Defaults:**
- **Edge nodes:** Drop `db.*`, `ncl.*`, `ncltransport.*`
- **Orchestrators:** Export all metrics (no filtering)

**Export all metrics** (for debugging):

```yaml
telemetry:
  drop_metric_prefixes: []  # Empty list keeps all metrics
```

**Add custom filters:**

```yaml
telemetry:
  drop_metric_prefixes:
    - "db."
    - "ncl."
    - "ncltransport."
    - "store_gc."  # Also drop garbage collection metrics
```

### Labels

```yaml
labels:
  region: us-west-2
  environment: production
  hardware: gpu
```

Labels are used for:
- Node selection in deployment targeting
- Filtering nodes in the workspace UI
- Grouping metrics and logs

Labels are also exposed to pipelines running on the node as Bloblang metadata under `@node_label_<key>` — see [Execution Context Metadata](/guides/pipelines/metadata#execution-context-metadata) for details.

## Environment Variables

All configuration can be overridden with environment variables:

```bash
# Core settings
export EXPANSO_EDGE_NAME=edge-node-01
export EXPANSO_EDGE_DATA_DIR=/var/lib/expanso/edge

# Workspace
export EXPANSO_EDGE_ORCHESTRATOR_ADDRESS=nats://orchestrator.expanso.io:4222

# Bootstrap TLS (for self-signed certificates)
export EXPANSO_EDGE_BOOTSTRAP_INSECURE=true  # Skip TLS verification

# API
export EXPANSO_EDGE_API_LISTEN_ADDR=localhost:9010

# Logging
export EXPANSO_EDGE_LOG_LEVEL=debug
export EXPANSO_EDGE_LOG_FORMAT=text
```

Environment variables take precedence over file configuration.

## Configuration File Locations

The edge node looks for configuration in these locations (in order):

1. Path specified with `--config` flag
2. `/etc/expanso/edge/config.yaml`

## Example Configurations

### Production Deployment

```yaml
name: edge-prod-${HOSTNAME}
data_dir: /var/lib/expanso/edge

orchestrator:
  address: nats://orchestrator.expanso.io:4222
  require_tls: true

api:
  listen_addr: localhost:9010
  auth:
    organization_id: org_abc123
    jwt:
      issuer: https://cloud.expanso.io

log:
  level: info
  format: json

telemetry:
  endpoint: collector.expanso.io:4317
  protocol: grpc
  resource_attributes:
    environment: production
    region: us-west-2

labels:
  environment: production
  region: us-west-2
```

### Development/Testing

```yaml
name: dev-edge
data_dir: ./dev-data

local_mode: true  # No workspace

api:
  listen_addr: localhost:9010

log:
  level: debug
  format: text

telemetry:
  do_not_track: true  # Disable telemetry
```

### High-Security Environment

```yaml
name: secure-edge-01
data_dir: /var/lib/expanso/edge

orchestrator:
  address: nats://orchestrator.internal:4222
  require_tls: true

api:
  listen_addr: localhost:9010  # Local-only
  auth:
    username: admin
    password: ${ADMIN_PASSWORD}

log:
  level: warn
  format: json

telemetry:
  endpoint: internal-collector:4317
  protocol: grpc
```

## Validation

The edge node validates configuration on startup:

- `data_dir` is required
- `log.level` must be valid option
- `log.format` must be valid option
- `telemetry.protocol` must be "grpc" or "http"

Invalid configuration will cause startup failure with clear error messages.

## Next Steps

- **[Installation Guide](/getting-started/installation)** - Install Expanso Edge
- **[Quick Start](/getting-started/quickstart/)** - Build your first pipeline

## Deploy to Your First Edge Node

# Deploy to Your First Edge Node

In the [Getting Started](/getting-started) tutorial, you ran both the orchestrator and edge node on your local machine. That's perfect for learning the basics, but production edge computing is different, and you'll deploy jobs to remote machines with real network challenges, firewall rules, and intermittent connectivity.

In this tutorial, you'll set up a real edge node on a separate Linux server or VM, configure it to connect to Expanso Cloud across a network, and deploy a data processing job that continues working even when network connectivity is disrupted. By the end, you'll understand how Expanso's edge architecture handles the realities of distributed computing.

This tutorial takes about 30-40 minutes to complete.

## Deploying as a Kubernetes Sidecar

If you're running workloads in Kubernetes, you can deploy Expanso Edge as a sidecar container alongside your existing application pods. This approach leverages standard Kubernetes patterns to collect logs, metrics, or other telemetry without managing separate edge servers.

:::note[Kubernetes Sidecar Example]
Replace the placeholder values:
- `YOUR_BOOTSTRAP_TOKEN`: a bootstrap token created in the Expanso Cloud console (Keys page).
- `your-app-image`: your application container image.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: your-app-image:latest
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        args: ["run"]
        env:
          - name: EXPANSO_EDGE_BOOTSTRAP_TOKEN
            value: YOUR_BOOTSTRAP_TOKEN
        volumeMounts:
          - name: expanso-cache
            mountPath: /var/lib/expanso
      volumes:
      - name: expanso-cache
        emptyDir: {}
```

:::

:::tip[Kubernetes Logging Best Practices]
- **Prefer stdout/stderr**: Kubernetes captures container stdout/stderr logs by default. Emitting logs there integrates with cluster logging drivers.
- **Use a shared volume for file logs**: If your app writes logs to files, mount them on an `emptyDir` (or `hostPath` for host-level logs) and point Expanso at those paths.
:::

## What You'll Learn

- How to prepare a remote Linux machine as an edge node
- How to install and configure the Expanso edge binary on a remote system
- How to set up network connectivity and authentication between nodes and your workspace
- How to use node labels for targeted job deployment
- How to monitor edge node health and connectivity
- How to test and verify autonomous operation during network issues
- How to troubleshoot common connectivity problems

## Prerequisites

Before starting, make sure you have:

- **An Expanso Cloud account and workspace** (sign up at [cloud.expanso.io](https://cloud.expanso.io))
- **A separate Linux machine or VM** for the edge node (Ubuntu 20.04+, Debian 11+, or similar)
- **SSH access** to the remote machine
- **Outbound network connectivity** from the edge node to Expanso Cloud
- **Basic familiarity** with Linux command line
- **Firewall configuration access** (if applicable)

:::tip[Cloud or Physical?]
You can use a cloud VM (AWS EC2, GCP Compute, Azure VM) or a physical server. The steps are identical. Expanso Edge has a minimal footprint and runs on virtually any modern hardware.
:::

## Step 1: Prepare Your Edge Machine

First, let's prepare your remote machine for Expanso.

**SSH into your edge machine:**

```bash
ssh user@your-edge-machine.example.com
```

**Verify system compatibility:**

```bash
# Verify Linux kernel version (3.10+ required)
uname -r

# Check available disk space for buffering
df -h /var/lib

# Verify network connectivity to your workspace
ping cloud.expanso.io
```

**Create the Expanso data directory:**

```bash
sudo mkdir -p /var/lib/expanso
sudo mkdir -p /etc/expanso
sudo chown $USER:$USER /var/lib/expanso /etc/expanso
```

This directory will store pipeline configurations, state, and buffered data when the node operates offline.

:::info[Why /var/lib/expanso?]
Following Linux filesystem hierarchy standards, `/var/lib/expanso` stores variable application data that persists across reboots. This is where buffered messages, local state, and temporary pipeline data live during normal operation and network outages.
:::

## Step 2: Install the Expanso Edge Binary

Now let's install the edge node software on your remote machine.

**Install the edge agent:**

```bash
curl -fsSL https://get.expanso.io/edge/install.sh | sudo bash
```

**Verify the installation:**

```bash
expanso-edge version
```

You should see output like:

```
Expanso Edge <version>
```

**Create a systemd service** (optional but recommended for production):

Download the service file:

```bash
sudo curl -o /etc/systemd/system/expanso-edge.service https://docs.expanso.io/examples/deployment/expanso-edge.service
sudo systemctl daemon-reload
```

Or [view the service file](pathname:///examples/deployment/expanso-edge.service)

We'll configure and start this service shortly, but having it defined now means your edge node will automatically restart if the machine reboots.

## Step 3: Configure Network Connectivity

Edge nodes connect outbound to Expanso Cloud: HTTPS on port 443 to `cloud.expanso.io` to fetch bootstrap credentials, then NATS on port 4222 to your workspace. Expanso Cloud hosts your workspace, so there is nothing to configure on the workspace side. You only need to make sure the edge node can reach Expanso Cloud.

**Test connectivity from the edge node:**

```bash
# Reach the bootstrap endpoint over HTTPS
nc -zv cloud.expanso.io 443

# You should see:
# Connection to cloud.expanso.io 443 port [tcp/*] succeeded!
```

The agent also needs outbound access to your workspace's NATS endpoint on port 4222.

:::caution[Cloud Security Groups]
If your edge node runs on a cloud VM, make sure its security groups or network ACLs allow outbound connections to Expanso Cloud on ports 443 and 4222. No inbound ports need to be opened on the edge node.
:::

## Step 4: Create a Bootstrap Token

Edge nodes authenticate to your workspace using a bootstrap token during initial registration. Bootstrap tokens are short-lived and are created in Expanso Cloud, not from the CLI.

**Create a token in the Expanso Cloud console:**

1. Open the Expanso Cloud console and go to the **Keys** page.
2. Create a new **bootstrap token**. Give it a description (for example, "Production edge node in Seattle datacenter") so you can identify it later.
3. Copy the token value. It looks like this:

```
exp_bk_a4b5c6d7e8f9g0h1i2j3k4l5
```

You'll only see the token value once, so copy it now, since you'll pass it to the agent in the next step.

:::info[Token Security]
Bootstrap tokens are short-lived and can be revoked. A single token can register one or more nodes while it is valid. Once a node registers successfully, it receives long-term credentials and no longer needs the bootstrap token. Keeping the validity window short limits the blast radius if a token is compromised.
:::

## Step 5: Configure the Edge Node

Now let's create a configuration file for your edge node with node identity, labels, and workspace connection details.

**Create the edge configuration file:**

Download the template configuration:

```bash
sudo curl -o /etc/expanso/edge-config.yaml https://docs.expanso.io/examples/deployment/edge-config.yaml
```

Or [view the configuration file](pathname:///examples/deployment/edge-config.yaml)

**Important:** Edit the file to customize:
- `name`: Your edge node's name (optional; defaults to the hostname)
- `labels`: Labels for job targeting (region, datacenter, environment, etc.)

Let's break down the key sections:

**Node identity:** The `name` is human-friendly but doesn't need to be globally unique. Your workspace will assign a unique ID during registration. The `labels` are crucial, since you'll use these to target jobs to specific nodes.

**Workspace connection:** You don't configure a workspace address. During bootstrap the edge node registers with Expanso Cloud and automatically receives its workspace endpoint and credentials. You only provide the bootstrap token, as a CLI flag or environment variable (not in the config file).

**Data directory:** The `data_dir` stores pipeline state and buffers messages during network outages.

:::info[Bootstrap Token Handling]
Bootstrap tokens should be provided via CLI flag (`--bootstrap-token`) or environment variable (`EXPANSO_EDGE_BOOTSTRAP_TOKEN`), not in configuration files. This ensures tokens are never accidentally committed to version control or logged.
:::

## Step 6: Start the Edge Node

With configuration in place, let's start the edge node and verify it connects successfully.

**Start the edge node:**

Pass the bootstrap token from Step 4 using the `--bootstrap-token` flag:

```bash
expanso-edge run \
  --config=/etc/expanso/edge-config.yaml \
  --bootstrap-token=exp_bk_a4b5c6d7e8f9g0h1i2j3k4l5
```

Or use an environment variable:

```bash
export EXPANSO_EDGE_BOOTSTRAP_TOKEN=exp_bk_a4b5c6d7e8f9g0h1i2j3k4l5
expanso-edge run --config=/etc/expanso/edge-config.yaml
```

Watch the output as the agent starts. It connects to your workspace over NATS, presents the bootstrap token, and registers. Once registration succeeds, the node receives long-term credentials in place of the bootstrap token, and the agent begins reporting a healthy state. The output resembles:

```
INFO  connecting to orchestrator over NATS
INFO  registering node with bootstrap token
INFO  registration complete, node connected and ready
```

Perfect! Your edge node is now registered and connected. Let's verify it registered using the CLI.

**In a new terminal on your local machine**, list nodes with the management CLI:

```bash
expanso-cli node list
```

You should see your new edge node. `node list` prints these columns: `ID`, `NAME`, `STATE`, `VERSION`, `LABELS`, and `RESOURCES`:

```
ID                NAME             STATE      VERSION   LABELS                                          RESOURCES
node-a1b2c3d4e5f6 edge-seattle-01  connected  <version>   datacenter=seattle,environment=production,...   4 CPU, 8 GB
```

**Get detailed node information:**

```bash
expanso-cli node describe node-a1b2c3d4e5f6
```

`node describe` prints four sections: `NODE DETAILS`, `SYSTEM INFORMATION`, `CAPABILITIES`, and `LABELS`:

```
NODE DETAILS
  ID:       node-a1b2c3d4e5f6
  Name:     edge-seattle-01
  State:    connected
  Version:  <version>

SYSTEM INFORMATION
  OS:            linux
  Architecture:  amd64
  CPU Cores:     4
  Memory:        8192 MB

CAPABILITIES
  Ready to run pipeline inputs, processors, and outputs

LABELS
  datacenter=seattle
  environment=production
  hardware=cpu
  region=us-west
```

The `State` field and the `LABELS` section are what you'll rely on most: `State` tells you the node is reachable and healthy, and the labels are what job selectors match against. Your edge node is fully operational.

:::tip[Systemd Service]
For production deployments, use the systemd service we created earlier. Stop the foreground process (Ctrl+C) and start the service:

```bash
sudo systemctl enable expanso-edge
sudo systemctl start expanso-edge
sudo systemctl status expanso-edge
```

This ensures the edge node starts automatically on boot and restarts if it crashes.
:::

## Step 7: Deploy a Job to the Edge Node

Now let's deploy a real data processing job that targets your specific edge node using labels.

**Create a job that processes syslog messages:**

Download the example job configuration:

```bash
curl -o edge-syslog-processor.yaml https://docs.expanso.io/examples/deployment/edge-syslog-processor.yaml
```

Or [view the job configuration](pathname:///examples/deployment/edge-syslog-processor.yaml)

**Important:** Edit the file to customize:
- `spec.selector.match_labels`: Adjust to match your node labels
- `config.output.http_client.url`: Your logging ingestion endpoint
- `config.output.http_client.headers.Authorization`: Your authentication token

This job does several important things:

- **Selective targeting:** Only deploys to nodes with labels `region=us-west` AND `environment=production`
- **Resilient processing:** Reads local syslog, parses structured data, and filters for important messages
- **Dual output:** Sends to central logging but also buffers locally if the network is down
- **Edge enrichment:** Adds node identity to messages, which is crucial for multi-site deployments

**Deploy the job:**

```bash
expanso-cli job deploy edge-syslog-processor.yaml
```

The CLI confirms that the `syslog-processor` job was created and scheduled to the matching node (`node-a1b2c3d4e5f6`).

**Verify the job is running on your edge node:**

```bash
expanso-cli job executions syslog-processor
```

The output shows one execution of `syslog-processor` running on `edge-seattle-01` (`node-a1b2c3d4e5f6`). The pipeline is now processing syslog messages on your edge node in real-time!

To follow the pipeline's logs, use:

```bash
expanso-cli job logs syslog-processor
```

## Step 8: Monitor Edge Node Connectivity

One of Expanso's key features is handling network disruptions gracefully. Your workspace tracks each node's connectivity through periodic heartbeats and reflects the result in the node's `STATE`.

**Check current state:**

The fastest way to see how your fleet is doing is the `STATE` column in `node list`:

```bash
expanso-cli node list
```

**Look at a single node in detail:**

```bash
expanso-cli node describe node-a1b2c3d4e5f6
```

The `State` field reflects the node's current connectivity: `connected` when your workspace is receiving heartbeats, `disconnected` after heartbeats stop arriving, and `lost` after an extended timeout without heartbeats. There's no separate event-stream or connection command. `node list` and `node describe` are how you observe status.

:::tip[Production Monitoring]
In production, poll `expanso-cli node list` on a schedule and alert when any node's `STATE` leaves `connected`/`healthy`. This gives you fast notification of connectivity problems without needing to watch individual nodes.
:::

## Step 9: Test Network Partition Scenarios

Now let's simulate real-world network issues and verify that your edge node continues operating autonomously.

**Scenario 1: Brief Network Interruption**

Temporarily block network connectivity from your edge node:

```bash
# On the edge machine
sudo iptables -A OUTPUT -p tcp --dport 4222 -j DROP
```

**Watch what happens:**

From your workspace, poll the node's state:

```bash
expanso-cli node describe node-a1b2c3d4e5f6
```

As heartbeats start to be missed, the node's `State` moves from `connected` to `disconnected`, and after an extended timeout without heartbeats it becomes `lost`. But here's the critical part: **the pipeline keeps running on the edge node**. The local buffering mechanism activates, storing processed syslog messages under the node's data directory until connectivity returns.

**Restore connectivity:**

```bash
# On the edge machine
sudo iptables -D OUTPUT -p tcp --dport 4222 -j DROP
```

Within a short time, the next `node describe` shows the `State` returning to `connected`/`healthy`. This was a brief interruption, so the node keeps the **same session**, it didn't restart, it reconnected. The buffered messages now flush to the central logging system.

**Scenario 2: Extended Network Outage**

For longer outages, the edge node starts a new session when connectivity returns. Let's test this:

```bash
# On the edge machine, block connectivity for 2 minutes
sudo iptables -A OUTPUT -p tcp --dport 4222 -j DROP
sleep 120
sudo iptables -D OUTPUT -p tcp --dport 4222 -j DROP
```

When the node reconnects after a prolonged disconnection, it begins a new session while keeping its existing node ID. Poll `expanso-cli node describe node-a1b2c3d4e5f6` and you'll see the `State` return to `connected`/`healthy`. Your workspace resynchronizes state and verifies that the job is still running correctly.

:::info[Session Continuity]
Edge nodes maintain their identity (node ID) across sessions. Sessions are logical operational periods, not tied to process lifetime. This design allows the system to track and correlate events while handling network realities.
:::

## Step 10: Verify Autonomous Operation

Let's verify that your edge node truly operates independently during network outages.

**Check the local buffer during an outage:**

The example pipeline is configured to fall back to a local directory when its remote destination is unreachable. While connectivity is blocked, SSH to your edge node and inspect that fallback directory (in this example, under the data directory at `/var/lib/expanso/buffer/`):

```bash
# On the edge machine
ls -lh /var/lib/expanso/buffer/
```

You'll see buffered data files accumulating during the outage. View their contents:

```bash
tail -5 /var/lib/expanso/buffer/syslog-*.jsonl
```

You'll see properly formatted, processed syslog entries, enriched with node identity by the pipeline:

```json
{"timestamp":"Oct 20 16:05:45","hostname":"edge-seattle-01","program":"systemd","message":"Started Daily apt download activities.","node_id":"node-a1b2c3d4e5f6","node_hostname":"edge-seattle-01","ingested_at":"2025-10-20T16:05:45Z"}
{"timestamp":"Oct 20 16:06:02","hostname":"edge-seattle-01","program":"kernel","message":"warning: CPU throttling detected","node_id":"node-a1b2c3d4e5f6","node_hostname":"edge-seattle-01","ingested_at":"2025-10-20T16:06:02Z"}
```

The pipeline continued processing data locally, even without workspace connectivity!

**Watch the buffer flush after reconnection:**

When connectivity returns, watch the buffer directory:

```bash
watch -n 2 'ls -lh /var/lib/expanso/buffer/ | tail -5'
```

You'll see files disappear as messages are sent to the central logging system. This demonstrates Expanso's edge-first architecture: **process data locally, sync when possible, never lose data**.

## Verification Checklist

Let's verify everything is working correctly:

- ✅ Edge node is installed on a separate Linux machine
- ✅ Network connectivity to your workspace is configured (firewall, security groups)
- ✅ Edge node successfully registered using a bootstrap token
- ✅ Long-term credentials are stored and the bootstrap token is consumed
- ✅ Node appears with a `connected`/`healthy` STATE in `expanso-cli node list`
- ✅ Labels are correctly configured and visible in `expanso-cli node describe`
- ✅ Job deployed successfully to edge node based on label selectors
- ✅ Pipeline is processing data (syslog messages)
- ✅ Node STATE stays `connected`/`healthy` under normal operation
- ✅ Network interruptions are handled gracefully (state moves to `disconnected`, then `lost`)
- ✅ Data buffers locally during outages
- ✅ Buffered data flushes when connectivity returns
- ✅ Node reconnects automatically after network issues

If all items are checked, congratulations! You have a production-ready edge deployment.

## What You Learned

You've accomplished a lot in this tutorial:

- ✅ Set up a production edge node on a remote Linux machine
- ✅ Configured network connectivity and firewall rules between your workspace and the edge node
- ✅ Used a bootstrap token for secure initial registration
- ✅ Configured node labels for targeted job deployment
- ✅ Deployed a real-world data processing pipeline (syslog processing)
- ✅ Monitored node health and connectivity via node state
- ✅ Tested network partition scenarios and verified autonomous operation
- ✅ Confirmed local buffering during outages and synchronization on recovery

## Key Concepts

**Bootstrap Tokens:** Short-lived credentials for initial node registration, created on the Keys page in the Expanso Cloud console. A token can register one or more nodes while it is valid. After registration, nodes receive long-term credentials that don't expire.

**Node Labels:** Key-value pairs attached to nodes (like `region=us-west`, `environment=production`) used by job selectors to control where jobs run. Labels are metadata, not security boundaries.

**Heartbeats:** Periodic health reports from edge nodes to your workspace. Your workspace uses heartbeats to track connectivity and update each node's `STATE`.

**Session Continuity:** Edge nodes maintain logical sessions that persist across brief network interruptions. A new session starts after an extended outage or a process restart, but the node identity remains constant.

**Local Buffering:** When network connectivity is lost, edge nodes buffer processed data locally. When connectivity returns, buffered data automatically syncs to remote destinations.

**Autonomous Operation:** Edge nodes continue processing data during network outages without workspace connectivity. Your workspace tracks desired state but doesn't need to be reachable for pipelines to function.

:::tip[Deep Dive]
Want to understand the architecture behind these features? Read:
:::

## Next Steps

Now that you have a production edge deployment, here's where to go next:

**Scale Your Deployment:**

**Advanced Job Configuration:**

**Production Operations:**

**Architecture Deep Dives:**

## Troubleshooting

### Edge Node Can't Connect to Your Workspace

**Symptom:** Edge node logs show connection errors or timeouts.

**Diagnosis:**

```bash
# On edge node, test connectivity
nc -zv cloud.expanso.io 443

# Check DNS resolution
nslookup cloud.expanso.io

# Verify routing
traceroute cloud.expanso.io
```

**Common Causes:**

1. **Firewall blocking traffic:** Verify firewall rules allow outbound TCP on port 4222
2. **Security groups (cloud):** Check cloud security group rules
3. **DNS issues:** Verify hostname resolves correctly
4. **NAT/routing:** Ensure a network route exists between the edge node and your workspace

**Solution:**

```bash
# From the edge node, confirm it can reach Expanso Cloud
nc -zv cloud.expanso.io 443

# Verify DNS resolves
getent hosts cloud.expanso.io

# If the edge node has an outbound firewall, allow outbound TCP to Expanso Cloud on ports 443 and 4222
```

### Bootstrap Token Authentication Failed

**Symptom:** Node logs show "invalid bootstrap token" or "token expired".

**Diagnosis:**

Open the **Keys** page in the Expanso Cloud console and check the token you used:

1. Confirm it hasn't expired (bootstrap tokens are short-lived).
2. Confirm it hasn't been revoked, and that it hasn't reached its use limit if one was set when the token was created.

**Common Causes:**

1. **Token expired:** Bootstrap tokens have short lifespans
2. **Token revoked or at its use limit:** the token was revoked, or it reached a maximum number of uses set when it was created
3. **Token typo:** Copy-paste errors when passing the token to the agent

**Solution:**

Create a fresh bootstrap token on the Keys page, then restart the agent with the new token:

```bash
# Restart with the new token via flag...
expanso-edge run \
  --config=/etc/expanso/edge-config.yaml \
  --bootstrap-token=exp_bk_NEW_TOKEN_VALUE

# ...or via environment variable, then restart the service
export EXPANSO_EDGE_BOOTSTRAP_TOKEN=exp_bk_NEW_TOKEN_VALUE
sudo systemctl restart expanso-edge
```

### Node Shows "Offline" Despite Running

**Symptom:** Edge node process is running but your workspace shows the node's STATE as "offline".

**Diagnosis:**

```bash
# Check edge node logs
sudo journalctl -u expanso-edge -f

# Check the node's state with the CLI
expanso-cli node describe <node-id>

# Check network connectivity
nc -zv cloud.expanso.io 443
```

**Common Causes:**

1. **Heartbeat timeout:** Network latency causing heartbeats to arrive late
2. **Clock skew:** System clocks out of sync between the edge node and your workspace
3. **NATS connection issues:** TLS handshake failures

**Solution:**

```bash
# Sync system clock (edge node)
sudo timedatectl set-ntp true

# Check TLS certificate validity (if using custom CA)
openssl s_client -connect cloud.expanso.io:443

sudo systemctl restart expanso-edge
```

### Job Deployed But Not Running on Edge Node

**Symptom:** Job shows as deployed but execution status is "pending" or "failed".

**Diagnosis:**

```bash
# Check job executions
expanso-cli job executions <job-name>

# View the pipeline's logs
expanso-cli job logs <job-name>

# Verify node capabilities and labels
expanso-cli node describe <node-id>
```

**Common Causes:**

1. **Label mismatch:** Job selector doesn't match node labels
2. **Resource constraints:** Node doesn't have required CPU/memory
3. **Missing dependencies:** Job requires unsupported inputs/outputs
4. **Configuration errors:** Invalid pipeline configuration

**Solution:**

```bash
# Verify label matching (node list shows LABELS by default)
expanso-cli node list

# Review the job's selector (job describe shows the selector by default)
expanso-cli job describe <job-name>

# Check execution details for errors
expanso-cli execution describe <execution-id>

# View agent logs on edge node
sudo journalctl -u expanso-edge -f | grep <job-name>

# Validate job configuration locally
expanso-cli job validate <job-file.yaml>
```

### Buffered Data Not Flushing After Reconnection

**Symptom:** Buffer files remain on disk after connectivity returns.

**Diagnosis:**

```bash
# Check buffer directory (on edge node)
ls -lh /var/lib/expanso/buffer/

# Verify output destination is reachable
curl -I https://logs.example.com/ingest

# Check pipeline logs
sudo journalctl -u expanso-edge -f | grep fallback
```

**Common Causes:**

1. **Destination unreachable:** Remote endpoint still down
2. **Authentication issues:** Credentials expired or invalid
3. **Rate limiting:** Remote service throttling requests
4. **Disk full:** No space for temporary files during flush

**Solution:**

```bash
# Test destination manually
curl -X POST https://logs.example.com/ingest \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"test": "message"}'

# Check disk space
df -h /var/lib/expanso

# Manually trigger flush (restart edge service)
sudo systemctl restart expanso-edge

# Increase flush rate in config if destination can handle it
# config.output.http_client.max_in_flight: 50
```

### Need More Help?

If you're still experiencing issues:

1. **Check the edge node logs:** Detailed logs are your best diagnostic tool
   ```bash
   sudo journalctl -u expanso-edge -n 100
   ```
   Your workspace is managed by Expanso Cloud. If the problem looks like it is on the workspace side, check the Cloud console or contact support.

2. **Enable debug logging:** Temporarily increase verbosity
   ```yaml
   # /etc/expanso/edge/config.yaml
   log:
     level: debug
   ```

3. **Community support:**
   - Email [support@expanso.io](mailto:support@expanso.io)
   - Ask in the [community Slack](https://exso.cloud/slack)

4. **File a bug report:**
   - Include logs from both your workspace and the edge node
   - Describe steps to reproduce
   - Share configuration files (redact secrets)

---

## Running Edge as a Systemd Service

# Running Edge as a Systemd Service

Running Expanso Edge as a systemd service is the recommended approach for production Linux deployments. Systemd handles automatic startup on boot, restarts if the process crashes, and integrates with standard Linux logging tools.

## Prerequisites

Before you start, make sure you have:

- A Linux system with systemd (most modern distributions)
- Root or sudo access
- Expanso Edge binary installed (see the [Installation](/getting-started/installation) guide)
- A bootstrap token from Expanso Cloud

## Directory Structure

Expanso Edge uses two main directories on Linux:

```
/etc/expanso/
├── edge-config.yaml      # Main configuration
└── bootstrap.env         # Bootstrap token (chmod 600)

/var/lib/expanso/
├── auth/                 # Credentials, auto-created after bootstrap
└── config.d/             # Dynamic config overrides
```

| Directory | Purpose |
|-----------|---------|
| `/etc/expanso/` | Configuration files, including the main config and environment file for secure token storage |
| `/var/lib/expanso/` | Runtime data: credentials created after bootstrap (in `auth/`) and dynamic configuration overrides |

## Setup Sequence

Follow these steps to set up Expanso Edge as a systemd service.

### Step 1: Create User and Directories

Create a dedicated system user and the required directories:

```bash
# Create system user with no login shell
sudo useradd -r -s /bin/false expanso

# Create directories
sudo mkdir -p /etc/expanso /var/lib/expanso

# Set ownership
sudo chown expanso:expanso /var/lib/expanso
```

:::info[Why a Dedicated User?]
Using a system user with no login shell improves security by limiting what the edge process can access.
:::

### Step 2: Install the Edge Binary

Install the edge agent with the official install script:

```bash
# Install the latest edge agent
curl -fsSL https://get.expanso.io/edge/install.sh | sudo bash

# Verify installation
expanso-edge version
```

The install script places a single statically-compiled binary with no external dependencies on your PATH.

### Step 3: Configure the Bootstrap Token

Store the bootstrap token in an environment file with restricted permissions:

```bash
# Create environment file (replace with your actual token)
echo "EXPANSO_EDGE_BOOTSTRAP_TOKEN=exp_bk_your_token_here" \
  | sudo tee /etc/expanso/bootstrap.env > /dev/null

# Restrict permissions to root only
sudo chmod 600 /etc/expanso/bootstrap.env
sudo chown root:root /etc/expanso/bootstrap.env
```

:::caution[Token Security]
Bootstrap tokens are time-limited credentials for initial node registration. Never store them in configuration files or commit them to version control. The environment file approach keeps the token readable only by root, and systemd loads it before starting the service.
:::

### Step 4: Run Initial Bootstrap

Register the node with Expanso Cloud:

```bash
sudo -u expanso expanso-edge bootstrap --token="$TOKEN"
```

This creates credentials in `/var/lib/expanso/auth/` that the service uses for subsequent connections. After bootstrap succeeds, the node receives long-term credentials and no longer needs the bootstrap token.

### Step 5: Create the Systemd Service

Create the service file at `/etc/systemd/system/expanso-edge.service`:

```ini
[Unit]
Description=Expanso Edge Node
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=expanso
Group=expanso
EnvironmentFile=-/etc/expanso/bootstrap.env
ExecStart=/usr/local/bin/expanso-edge run \
  --config=/etc/expanso/edge-config.yaml \
  --data-dir=/var/lib/expanso
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

:::tip[Service File Options]
- **`EnvironmentFile=-`**: The `-` prefix makes the file optional—if it doesn't exist, systemd won't fail to start the service
- **`Restart=always`**: Automatically restarts the service if it exits for any reason
- **`RestartSec=10`**: Waits 10 seconds before restarting to avoid rapid restart loops
:::

### Step 6: Enable and Start the Service

```bash
# Reload systemd to pick up the new service file
sudo systemctl daemon-reload

# Enable the service to start on boot
sudo systemctl enable --now expanso-edge

# Check status
sudo systemctl status expanso-edge
```

The service will now start automatically on boot and restart if it crashes.

## Custom Bootstrap URL

For self-hosted or on-premises control planes, specify a custom bootstrap URL instead of the default Expanso Cloud endpoint.

**During initial bootstrap:**

```bash
expanso-edge bootstrap --token "$TOKEN" --url "https://bootstrap.mycompany.com"
```

**In the run command (for ongoing use):**

```bash
expanso-edge run --bootstrap-url "https://bootstrap.mycompany.com"
```

This is useful for air-gapped environments or organizations running their own Expanso control plane.

## Viewing Logs

Expanso Edge integrates with journald when running as a systemd service:

| Location | Purpose |
|----------|---------|
| `journalctl -u expanso-edge -f` | Primary logs (systemd journal) |
| `/var/log/expanso-edge.log` | Optional file output (set in config) |

**View recent logs:**

```bash
sudo journalctl -u expanso-edge -n 100
```

**Follow logs in real-time:**

```bash
sudo journalctl -u expanso-edge -f
```

**Filter by time range:**

```bash
sudo journalctl -u expanso-edge --since "1 hour ago"
```

:::info[Offline Buffering]
When the edge node can't reach its destinations, it buffers data locally and forwards it once connectivity returns.
:::

## Managing the Service

Use standard systemctl commands to manage the edge node:

```bash
# Check status
sudo systemctl status expanso-edge

# Restart
sudo systemctl restart expanso-edge

# Stop for maintenance
sudo systemctl stop expanso-edge

# View recent logs
sudo journalctl -u expanso-edge -n 50
```

## What's Next?

- **[Edge Node Configuration](/operations/configuration)**: Configure logging, metrics, and performance tuning
- **[Edge Deployment](/operations/deployment/edge-deployment)**: Deploy pipelines to your edge nodes
- **[Monitoring](/operations/monitoring)**: Set up monitoring and observability

## Job Deployments with Health Checks and Rollback

# Job Deployments with Health Checks and Rollback

For the Cloud console workflow, including Rolling and Immediate strategies,
Wave Size, pause/resume, version comparison, and rollback, start with
[Expanso Cloud deployments](/operations/cloud/deployments/).

When you update jobs running on edge nodes, Expanso validates new versions before fully committing to them. If health checks fail, your workspace automatically rolls back to the previous version to keep your edge infrastructure stable.

This guide explains how job deployments work, how to configure health checks, and how the rollback system keeps your edge infrastructure resilient.

## Overview

When you deploy a standard job, the new version starts immediately. **Deployment jobs** work differently—they enter a validation period with health checks before becoming fully active. If validation fails, Expanso automatically reverts to the last working version.

This gives you automatic rollback when deployments fail, prevents bad deployments from destabilizing your edge fleet, and keeps things running even during failed updates.

## Job State Lifecycle

When you deploy a job, it moves through different states as it rolls out to your edge fleet and starts processing data. Knowing these states helps you monitor deployments and debug issues when things go wrong.

### Job States vs. Execution States

Expanso tracks two different kinds of states, and it's important to understand the difference:

- **Job states** show the overall status of your job across your entire fleet (like `deploying`, `running`, or `degraded`)
- **Execution states** show what's happening with individual pipeline instances on specific nodes (like `starting`, `validating`, or `running`)

:::info[Single Source of Truth]
The job state is what matters for monitoring; it reflects the deployment and health status of your job across all edge nodes. Execution states are implementation details that your workspace uses internally.
:::

### Job State Transitions

When you deploy or update a job, it flows through these states:

```
pending → queued → deploying → running
                       ↓
                   rollout_paused (manual pause)
                       ↓
                   rollout_failed (health checks failed)
```

After a job reaches `running`, it can transition to `degraded` if health issues pop up:

```
running ↔ degraded (daemon jobs only)
```

### State Descriptions

Here's what each state means and when you'll see it.

**pending**: Your job is created but hasn't been scheduled yet. Your workspace is preparing to start the deployment.

**queued**: The job is scheduled but there aren't any nodes available to run it. You'll see this when your fleet is at capacity or when no nodes match the job's selector labels.

**deploying**: The rollout is actively happening. Your workspace is deploying the job to nodes, and executions are going through health validation. 

During this state, the job is eligible for automatic rollback if health checks fail. This is the critical window where Expanso validates your deployment before it goes fully live.

**running**: Everything's healthy and stable. The rollout completed successfully, and your job is processing data normally across the fleet.

**rollout_paused**: You manually paused the rollout. The deployment is frozen—no new nodes get updated, but nodes that are already running the job keep working. 

You can resume the rollout whenever you're ready, or roll back to the previous version if something looks wrong.

**rollout_failed**: Health checks failed during deployment, and Expanso automatically rolled back the job. It won't make further deployment progress until you fix the issue and redeploy.

This happens when your workspace detects that executions aren't passing health validation during the `deploying` state.

**degraded**: Your job was running fine, but some executions have become unhealthy. This only applies to daemon-type jobs that continuously retry failed executions.

Your workspace monitors degraded jobs but doesn't automatically roll back, since the deployment was previously validated. Degradation usually means there's an issue with specific nodes or data sources, not the job configuration itself.

**completed**: The job finished successfully. This only applies to batch-type jobs that have a defined end state.

**failed**: The job failed—typically this means all executions failed or the job hit a terminal error it couldn't recover from.

**stopped**: You explicitly stopped the job using the API or CLI. The job won't restart until you start it again.

### Rollout Completion Semantics

Here's a gotcha to watch out for: the job's `status.rollout.completed_at` timestamp tells you when a rollout finished—not the job state.

:::caution[Checking Rollout Status]
A rollout is complete when `completed_at` has a non-null value, regardless of whether the job state is `running`, `degraded`, or `rollout_failed`.

When you're checking if a rollout is still active, use `completed_at == null` to detect it—don't rely on the job state.
:::

### Daemon vs. Ops Jobs

State transitions work differently depending on your job type.

**Daemon jobs** run continuously and process data indefinitely:
- They can transition between `running` and `degraded` based on ongoing health
- Edge nodes retry failed executions with exponential backoff
- Your workspace monitors degraded daemon jobs but doesn't immediately roll them back

**Ops jobs** are one-time operations that finish when the work is done:
- They transition from `running` to `completed` or `failed` when finished
- They don't use the `degraded` state since they're not continuously retrying
- There's no automatic rollback after completion—if you need to undo changes, use manual rollback

## How Deployments Work

### Execution State Lifecycle

When you deploy a new version of a job with deployment configuration, executions go through an extended lifecycle:

```
Pending → Starting → Validating → Running
                         ↓
                      Failed → Workspace Rollback
                         ↓
                      Degraded (retrying)
```

Here's what each state means:

**Pending** means the execution is scheduled but not started yet. **Starting** means it's being initialized. **Validating** is where things get interesting—the execution is running but undergoing health validation, so it's still eligible for rollback. **Running** means the execution passed health checks and is stable. **Failed** means the execution is terminal (typically for ops-type pipelines). **Degraded** means the execution is unhealthy but retrying (typically for daemon-type pipelines).

The **Validating** state is unique to deployments. During this window, health checks evaluate the execution using consecutive time intervals, a deadline limits how long validation can take, and if validation fails, your workspace initiates rollback. Once an execution reaches Running state, it's marked as stable.

When a daemon-type pipeline encounters failures, edge nodes mark the execution as **Degraded** and retry locally with backoff. Your workspace monitors these Degraded executions, and if too many nodes fail, it triggers a coordinated rollback across your fleet.

### Health Check Configuration

When you deploy a new job, you want to make sure it's actually working before it takes over from the previous version. Health checks let Expanso watch your deployment as it starts up, validate that it's processing data without errors, and automatically roll back if something goes wrong.

Expanso uses window-based health evaluation—it watches your job over consecutive time intervals and counts how many are healthy vs. unhealthy. This prevents both false positives (one lucky success) and false negatives (one transient error) from affecting your deployment.

Here's how to configure them:

```yaml
name: my-api-service
type: pipeline
config:
  input: { http_server: { path: /ingest } }
  output: { stdout: {} }
rollout:
  strategy: rolling
  health_check:
    interval: 10s              # Evaluation window duration (default: 10s)
    success_threshold: 2       # Consecutive healthy windows needed (default: 2)
    failure_threshold: 3       # Consecutive unhealthy windows before rollback (default: 3)
    max_error_rate: 0.10       # Maximum error rate per window (default: 0.10 = 10%)
    deadline: 5m               # Maximum time to wait for validation
  # Other rollout options...
selector:
  match_labels:
    env: production
```

**interval** (default: 10s) sets how long each health evaluation window lasts. The system calculates error rates per interval, not over the lifetime of the execution.

**success_threshold** (default: 2) is how many consecutive healthy intervals you need before the execution is validated and marked as stable. This prevents transient successes from passing validation.

**failure_threshold** (default: 3) is how many consecutive unhealthy intervals trigger a rollback. This prevents transient errors from causing unnecessary rollbacks.

**max_error_rate** (default: 0.10) sets the maximum error rate allowed within an interval. If errors exceed 10% of processed messages in a window, that window is marked unhealthy.

**deadline** (default varies) sets the maximum time allowed for the execution to achieve validation. If this expires without reaching the success threshold, the execution fails and triggers rollback.

#### How It Works

When your execution starts, it enters the **Starting** state. Once it begins running, it transitions to **Validating**.

During validation, the health check evaluates each interval. If the error rate is below **max_error_rate**, the window is healthy. Otherwise it's unhealthy. Idle windows with no traffic don't count toward either threshold.

After **success_threshold** consecutive healthy windows, the execution transitions to **Running** (stable). After **failure_threshold** consecutive unhealthy windows, it's marked as **Failed** and triggers rollback.

If the **deadline** expires without validation, the execution is also marked as **Failed** and rolls back.

### How Health Evaluation Works

During validation, Expanso checks each pipeline component separately to pinpoint exactly what's failing. Instead of getting a vague "pipeline unhealthy" message, you'll know which specific input, processor, or output has a problem.

#### Component-by-Component Evaluation

Expanso applies different health checks depending on the component type:

**Inputs** (e.g., `input.kafka.0`, `input.http_server.0`):
- Checks connection errors: `input_connection_failed` and `input_connection_lost` metrics
- Connection failures mark the input as unhealthy
- Error rate checks don't apply—inputs receive data, they don't process it

**Processors** (e.g., `processor.mapping.0`, `processor.filter.1`):
- Checks error rate: ratio of `processor_error` to `processor_received` messages
- If error rate exceeds `max_error_rate` threshold, the processor is unhealthy
- Connection checks don't apply—processors don't maintain connections

**Outputs** (e.g., `output.kafka.0`, `output.http_client.0`):
- Checks both connection errors AND error rate
- Connection metrics: `output_connection_failed` and `output_connection_lost`
- Error rate: `output_error` / `output_sent`
- Either connection failures or high error rate marks the output as unhealthy

#### Fail-Fast Strategy

Expanso checks components in order and stops at the first unhealthy one. This fail-fast approach gives you:

- **Precise diagnostics**: You know exactly which component failed (e.g., "Kafka Output [kafka]: connection failed")
- **Early detection**: No need to wait for all components to be evaluated
- **Clear root cause**: Instead of "pipeline unhealthy," you get "Data Filter [bloblang]: error rate 15.2% exceeds threshold 10.0%"

#### Activity-Based Evaluation

Components with no traffic during a window become Pending (not Unhealthy). This makes sense during:

- **Startup**: Pipeline is initializing, connections are establishing
- **Low traffic**: Waiting for data to arrive
- **Idle intervals**: Between bursts of activity

Pending windows don't count toward consecutive healthy or unhealthy thresholds. This prevents idle periods from triggering unnecessary rollbacks.

#### Health Check Lifecycle

For each interval window (default: 10 seconds):

1. **Metrics collection**: OTel exporter forwards component-specific metrics to the health tracker
2. **Window evaluation**: When the interval ends, Expanso evaluates each component:
   - Check for activity (any traffic?)
   - Check connection errors (inputs and outputs)
   - Check error rate (processors and outputs)
3. **Consecutive tracking**: 
   - Healthy window → increment consecutive healthy count, reset unhealthy count
   - Unhealthy window → increment consecutive unhealthy count, reset healthy count
   - Pending window → don't change either count
4. **State transition**:
   - If consecutive healthy count reaches `success_threshold` → transition to Running (stable)
   - If consecutive unhealthy count reaches `failure_threshold` → transition to Failed (triggers rollback)
   - If deadline expires → transition to Failed (triggers rollback)

## Continuous Health Monitoring

Health monitoring doesn't stop after your deployment passes initial validation. **Expanso continuously monitors health throughout the entire execution lifecycle**—whether the execution is in Validating, Running, or Degraded state.

This continuous approach ensures that stable deployments stay tracked for health degradation, late-joining nodes get properly validated before being marked as stable, and degraded executions must prove they're healthy before returning to Running state.

### How Continuous Monitoring Works

**During Validating State:**

When an execution is first validating, health checks evaluate it using the configured deadline. If health is proven (HealthHealthy), the execution transitions to Running and gets a StableAt timestamp. If health fails (HealthUnhealthy), the execution transitions to Degraded for daemon jobs or Failed for ops jobs.

Here's where it gets interesting: **if the deadline expires while health is still HealthPending**, the system gives "benefit of the doubt" and marks the execution as Running. This deadline-based benefit of doubt helps deployment waves progress even when data traffic is low or health checks haven't reported yet.

**During Running State (Stable):**

Once an execution has a StableAt timestamp, it's considered stable—but Expanso keeps monitoring. If health becomes HealthUnhealthy, the execution transitions to Degraded. **There's no deadline here**—the execution stays Running until it's proven unhealthy. This prevents stable deployments from being unnecessarily marked as degraded during idle periods.

**During Running State (Late-Joiners):**

Late-joiners are nodes that come online after the initial deployment. These executions don't have a StableAt timestamp yet, and they're handled differently. **They wait indefinitely for health to be proven**—there's no deadline pressure and no "benefit of doubt" for late-joiners. They must achieve HealthHealthy status before getting a StableAt timestamp. This ensures late-arriving nodes don't get marked as stable without validation.

**During Degraded State:**

Degraded executions must prove HealthHealthy to transition back to Running. **There's no "benefit of doubt" for recovery**—health must be explicitly proven. If health remains HealthUnhealthy or HealthPending, the execution stays Degraded. Edge nodes continue retrying with exponential backoff while reporting Degraded state, preventing executions from bouncing between states without actual recovery.

### Late-Joiner Behavior

A "late-joiner" is an edge node that comes online after a job deployment has already rolled out to other nodes. This commonly happens when you add a new edge node to your fleet after a job is deployed, an existing node was offline during deployment and reconnects later, or an edge node restarts and needs to catch up with the current job version.

**How Late-Joiners Are Handled:**

When a late-joiner execution starts, it's placed directly into the Running state (bypassing Validating). However, it doesn't immediately get a StableAt timestamp—it must prove health first.

The key difference from initial deployment validation is that **late-joiners wait indefinitely for health proof**. There's no deadline, and there's no "benefit of doubt" if health data is pending. The execution must explicitly achieve HealthHealthy status before receiving a StableAt timestamp. This conservative approach ensures that late-arriving nodes are fully validated before being considered part of your stable fleet, even if they join long after the initial deployment wave.

**Example Timeline:**

Here's what this looks like in practice. Job v2 is deployed at 10:00 AM to 50 nodes. All 50 nodes validate health and reach Running (stable) state by 10:05 AM. A new node joins your fleet at 2:00 PM and receives Job v2. The new node starts the execution in Running state (no StableAt yet), and the system continuously monitors health without any deadline. Once HealthHealthy is confirmed, the execution gets its StableAt timestamp, and the node is now fully validated and part of your stable deployment.

### State Transition Summary

Here's how executions transition between states based on health monitoring:

| Current State | Health Status | Deadline Expired? | Transition |
|--------------|---------------|-------------------|------------|
| Validating | HealthHealthy | N/A | → Running + StableAt |
| Validating | HealthUnhealthy | N/A | → Degraded (daemon) or Failed (ops) |
| Validating | HealthPending | Yes | → Running + StableAt (benefit of doubt) |
| Validating | No Runtime | Yes | → Degraded (daemon) or Failed (ops) |
| Running (stable) | HealthUnhealthy | N/A | → Degraded |
| Running (stable) | HealthHealthy or HealthPending | N/A | No change (stays Running) |
| Running (late-joiner) | HealthHealthy | N/A | Set StableAt (now fully stable) |
| Running (late-joiner) | HealthUnhealthy | N/A | → Degraded |
| Running (late-joiner) | HealthPending | N/A | Wait indefinitely (no benefit of doubt) |
| Degraded | HealthHealthy | N/A | → Running |
| Degraded | HealthUnhealthy or HealthPending | N/A | Stay Degraded (no benefit of doubt) |

:::tip[Understanding Deadlines and Benefit of Doubt]
The deadline only applies during the Validating state to help initial deployment waves progress. The "benefit of doubt" is only given during Validating—allowing low-traffic deployments to proceed when health data hasn't arrived yet. Late-joiners and degraded executions must prove health explicitly before transitioning to a stable state.
:::

### New Nodes During Active Deployments

When you scale your edge infrastructure by adding new nodes, the scheduler needs to decide whether to deploy jobs to them. This decision depends on the current deployment state.

The behavior here is different from late-joiners—nodes that come online after a deployment has already finished. During an active deployment, the scheduler protects you from accidentally deploying problematic versions to fresh capacity.

#### When New Nodes Get Jobs

Your new nodes will receive job deployments in most cases:

- **No active deployment**: Normal scheduling applies. New nodes get the current job version immediately.
- **In-progress deployment**: New nodes join the deployment and receive the new version as part of the rollout.
- **Completed deployment**: The deployment is done, so normal scheduling resumes.
- **Cancelled deployment**: The deployment was aborted, so normal scheduling resumes.

In these states, your infrastructure scales normally. New capacity comes online without manual intervention, and you don't need to do anything special.

#### When New Nodes Are Blocked

There are two deployment states where new nodes will NOT receive job deployments:

- **Paused deployment**: The deployment is frozen, waiting for your action. New nodes remain idle until you resume or cancel the deployment.
- **Failed deployment**: The deployment failed health checks and triggered a rollback. New nodes remain idle until you resolve the issue and resume or cancel the deployment.

This blocking behavior is intentional—it prevents new nodes from receiving potentially unstable job versions.

Here's why this matters: if you're investigating a deployment issue or waiting for a maintenance window, you don't want fresh capacity automatically picking up a problematic version. The scheduler holds back new nodes until you explicitly decide how to proceed.

#### Unblocking New Nodes

If you add nodes during a paused or failed deployment, you'll need to take action to unblock them:

1. **Resume the rollout** if you want to continue rolling out the new version:
   ```bash
   curl -X POST https://cloud.expanso.io/api/v1/jobs/{job-id}/rollout/resume \
     -H "Authorization: Bearer $TOKEN"
   ```

2. **Roll back** if you want to revert all nodes (including new ones) to the previous version:
   ```bash
   curl -X POST https://cloud.expanso.io/api/v1/jobs/{job-id}/rollback \
     -H "Authorization: Bearer $TOKEN"
   ```

:::tip[Monitoring Blocked Nodes]
Check your fleet status to see if nodes are waiting for jobs. Idle nodes during an active deployment might indicate a paused or failed state that needs attention.
:::

#### Job State Summary

The scheduler's behavior for new nodes depends on the current job state:

| Job State | New Nodes Get Job? | What To Do |
|-----------|-------------------|------------|
| `pending`, `queued`, `running`, `completed`, `stopped` | ✅ Yes | Nothing—normal scheduling |
| `deploying` | ✅ Yes | Nothing—nodes join the rollout |
| `rollout_paused` | ❌ No | Resume or roll back the rollout |
| `rollout_failed` | ❌ No | Investigate, then redeploy or roll back |
| `degraded` | ✅ Yes | Monitor—job is running but unhealthy |

When a job is in `rollout_paused` or `rollout_failed` state, the scheduler blocks new nodes from receiving the job. This prevents potentially unstable versions from deploying to fresh capacity while you're investigating issues.

## Execution States and Failure Handling

When deployments fail health checks, edge nodes report their execution state to your workspace. Understanding these states helps you monitor deployments and troubleshoot issues.

### Execution States

**Running**: The execution passed health validation and is stable. This is the healthy, steady state for a deployment.

**Degraded**: The execution is unhealthy but retrying. This typically happens with daemon-type pipelines (like continuous data processing) that can recover from transient failures. Edge nodes retry locally with 30-second backoff intervals, giving the pipeline time to recover.

**Failed**: The execution is terminal and won't retry. This typically happens with ops-type pipelines or when your workspace explicitly stops a failed deployment.

**Validating**: The execution is running but still undergoing health checks. It hasn't yet proven stable, so it's still eligible for rollback if health checks fail.

### How Edge Nodes Handle Failures

Edge nodes don't decide when to roll back; they report state to your workspace, which makes all rollback decisions.

When a daemon pipeline fails health checks, the edge node marks it as **Degraded** and retries locally with exponential backoff (starting at 30 seconds). The pipeline keeps attempting to recover while reporting its degraded state to your workspace.

When an ops pipeline fails, the edge node marks it as **Failed** (terminal state) and reports this to your workspace. There's no local retry since ops pipelines are typically one-time operations.

### How Your Workspace Decides Rollback

Your workspace monitors execution states across your entire edge fleet. When calculating deployment health, it counts both **Failed** and **Degraded** executions as failures.

If the number of failed/degraded nodes exceeds your deployment's health threshold, your workspace initiates a coordinated rollback. Your workspace atomically:

1. **Updates the job spec** to the previous version's configuration
2. **Cancels the failed deployment** (marked as failed with rollback message)
3. **Creates a new deployment** with immediate strategy using the rolled-back spec
4. **Increments the job version** (e.g., v1 → v2 fails → v3 with v1's spec)

This coordinated approach ensures all nodes return to a known-good configuration together, even if some nodes are retrying locally.

:::info[Rollback Prevention]
The system tracks when executions become stable to prevent rollback loops. When validation succeeds, the execution gets a `StableAt` timestamp. If a previously-validated execution degrades later, it won't trigger another rollback—it uses normal retry instead.

This prevents scenarios where version A is stable, version B fails and rolls back (creating v3 with A's spec), and then v3 also fails, which would otherwise trigger endless rollback loops.
:::

### First Deployment Behavior

For the first deployment of a job (when no previous version exists), the rollback behavior is different. If the deployment fails validation, your workspace stops the execution since there's no healthy version to roll back to.

This prevents infinite retry loops when the first deployment fails validation. If there's no healthy version, the system stops rather than looping.

## Manual Deployment Control

Expanso automatically handles deployment validation and rollback, but you can also take manual control through **job-scoped APIs**. Just provide the job name or ID—no deployment ID needed. You can pause, resume, or roll back deployments on demand.

### When to Use Manual Controls

**Pause** stops a rolling deployment temporarily. The current wave freezes—no new nodes get updated, but already-updated nodes keep running. Use this to investigate issues, coordinate with other systems, or wait for a maintenance window.

**Resume** continues a paused rollout. The deployment picks up where it left off and progresses through the remaining waves.

**Roll back** reverts to the previous version immediately. This is useful when you spot critical issues before automatic health checks trigger, or when you want to roll back during a maintenance window to minimize user impact.

### Job-Scoped Rollout APIs

Expanso provides job-scoped APIs to control rollouts. You only need the job name:

- `POST /jobs/{id}/rollout/pause`
- `POST /jobs/{id}/rollout/resume`
- `POST /jobs/{id}/rollback`

All endpoints require authentication and support dry-run mode and reason tracking.

### Pausing a Rollout

Pause an active rollout to temporarily stop its progress:

```bash
curl -X POST https://cloud.expanso.io/api/v1/jobs/my-job/rollout/pause \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Pausing for investigation"
  }'
```

Response:

```json
{
  "job_id": "job-abc123",
  "deployment_id": "dep-xyz789"
}
```

**Dry run mode** lets you check if the operation would succeed without actually pausing:

```bash
curl -X POST https://cloud.expanso.io/api/v1/jobs/my-job/rollout/pause \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Checking if pause is possible",
    "dry_run": true
  }'
```

**Error cases:**
- **409 Conflict**: No active deployment to pause, or deployment is already paused
- **404 Not Found**: Job doesn't exist
- **400 Bad Request**: Deployment is in a terminal state (completed, failed, canceled)

**Via CLI:**

You can also pause using the CLI:

```bash
# Pause rollout by job name
expanso-cli job pause-rollout my-job

# Pause with a reason
expanso-cli job pause-rollout my-job --reason "investigating performance issue"

# Dry run to check if pause is possible
expanso-cli job pause-rollout my-job --dry-run
```

### Resuming a Deployment

Resume a paused deployment to continue the rollout:

```bash
curl -X POST https://cloud.expanso.io/api/v1/jobs/my-job/rollout/resume \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Investigation complete, resuming rollout"
  }'
```

Response:

```json
{
  "job_id": "job-abc123",
  "deployment_id": "dep-xyz789"
}
```

The deployment continues from where it was paused—if it was in the middle of wave 2, it completes wave 2 and then proceeds to wave 3.

**Via CLI:**

You can also resume using the CLI:

```bash
# Resume rollout by job name
expanso-cli job resume-rollout my-job

# Resume with a reason
expanso-cli job resume-rollout my-job --reason "issue resolved, continuing deployment"

# Dry run to check if resume is possible
expanso-cli job resume-rollout my-job --dry-run
```

### Rolling Back a Job

Roll back a job to a previous version. This cancels any active deployment and creates a new immediate deployment with the previous version's configuration:

```bash
# Roll back to previous version (automatic)
curl -X POST https://cloud.expanso.io/api/v1/jobs/my-job/rollback \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Manual rollback due to increased error rate"
  }'
```

Response:

```json
{
  "job_id": "job-abc123",
  "deployment_id": "dep-new456",
  "from_version": 3,
  "to_version": 4,
  "target_spec": 2,
  "created_new": true,
  "cancelled_prior": true,
  "warnings": []
}
```

**Roll back to a specific version** by providing the version number:

```bash
curl -X POST "https://cloud.expanso.io/api/v1/jobs/my-job/rollback?version=1" \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Rolling back to last known good version"
  }'
```

**Understanding the response:**

| Field | Description |
|-------|-------------|
| `from_version` | The version you're rolling back from (e.g., 3) |
| `to_version` | The new version number after rollback (e.g., 4) - increments because the job spec is updated |
| `target_spec` | The version whose configuration you're rolling back to (e.g., 2) |
| `created_new` | Always true - rollback creates a new immediate deployment |
| `cancelled_prior` | True if there was an active deployment that got canceled |
| `warnings` | Any non-fatal issues encountered |

**Version semantics:**

When you roll back from version 3 to version 2's configuration:
1. Current deployment (version 3) is canceled
2. Job spec is reverted to version 2's configuration
3. Job version increments to 4 (new spec version)
4. New immediate deployment is created (version 4 with version 2's config)

This means "rollback" creates a new version with the old configuration, not a true rewind.

**Error cases:**
- **400 Bad Request**: Already at first version (no previous version to roll back to)
- **404 Not Found**: Target version doesn't exist in version history
- **409 Conflict**: Cannot roll back in current state

### Common Patterns

**Pause for investigation:**

```bash
# 1. Pause rollout
curl -X POST https://cloud.expanso.io/api/v1/jobs/api-service/rollout/pause \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -d '{"reason": "Investigating error spike"}'

# 2. Investigate (check logs, metrics, etc.)
expanso-cli job logs api-service

# 3. Resume if OK, or roll back if not
curl -X POST https://cloud.expanso.io/api/v1/jobs/api-service/rollout/resume \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -d '{"reason": "False alarm, continuing rollout"}'
```

**Emergency rollback:**

```bash
# Immediately roll back when you spot critical issues
curl -X POST https://cloud.expanso.io/api/v1/jobs/payment-processor/rollback \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -d '{"reason": "Critical: payment failures detected"}'
```

### Manual vs Automatic Control

**When to use manual control:**
- You spot critical issues before automatic health checks fail
- You want to control rollout timing during a maintenance window
- You're seeing early warning signs (increased latency, memory usage)
- You need immediate action for business reasons
- You want to pause to coordinate with other systems

**When to rely on automatic behavior:**
- Normal deployment operations with proper health check configuration
- Rolling out to large fleets where manual intervention isn't practical
- When you want consistent behavior based on objective health metrics
- During off-hours when no one is actively monitoring

## Comparing Job Versions

Before rolling back or investigating issues, you'll often want to see exactly what changed between versions. The versions diff endpoint lets you compare any two versions of a job and view a unified diff of the configuration changes.

### When to Compare Versions

**Before rolling back.** You're about to roll back from version 3 to version 2, but you want to verify what you're reverting. Comparing versions shows you exactly what configuration was added, removed, or modified.

**Auditing historical changes.** You need to understand how a job's configuration evolved over time. Comparing versions helps you trace when specific settings were introduced or removed.

**Debugging issues.** Your job was working fine at version 2 but broke at version 3. Comparing those versions helps you pinpoint the exact configuration change that caused the problem.

### Comparing Versions via API

Compare two specific versions:

```bash
curl -X GET "https://cloud.expanso.io/api/v1/jobs/my-job/versions/diff?from=1&to=3" \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN"
```

Compare a version against the latest:

```bash
curl -X GET "https://cloud.expanso.io/api/v1/jobs/my-job/versions/diff?from=1" \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN"
```

If you omit `to`, it defaults to the job's current version.

**Response:**

```json
{
  "job_id": "job-abc123",
  "from_version": 1,
  "to_version": 3,
  "diff": "--- version 1\n+++ version 3\n@@ -1,5 +1,6 @@\n name: my-job\n type: pipeline\n+config:\n+  input:\n..."
}
```

| Field | Description |
|-------|-------------|
| `job_id` | ID of the job |
| `from_version` | Base version compared |
| `to_version` | Target version compared |
| `diff` | Unified diff output showing the changes |

### Comparing Versions via CLI

```bash
# Compare version 1 to version 3
expanso-cli job versions-diff my-job --from 1 --to 3

# Compare version 1 to the latest version
expanso-cli job versions-diff my-job --from 1

# Compare versions in a specific namespace
expanso-cli job versions-diff my-job --from 1 --to 3 --namespace production
```

The CLI outputs the diff directly to your terminal, making it easy to see what changed at a glance.

## Manual Rollback

Something's wrong with your deployment. Maybe you're seeing increased error rates, or users are reporting issues—and you don't want to wait for automatic rollback thresholds to kick in. You can manually trigger a rollback using the Expanso API or CLI to immediately revert to the previous version.

### When to Manually Roll Back

**You're seeing critical issues before automatic thresholds trigger.** Your monitoring shows a spike in errors or performance degradation right after a deployment. Rather than wait for automatic health checks to fail, you want to roll back immediately to minimize impact.

**Early warning signs indicate problems.** You notice unusual patterns—increased latency, memory usage climbing, or warning messages in logs. The deployment hasn't technically failed health checks yet, but you want to roll back during your maintenance window before users are affected.

### Triggering a Rollback

Let's walk through how to trigger a rollback using both the API and CLI.

**Via API:**

```bash
curl -X POST https://cloud.expanso.io/api/v1/jobs/{job-id}/rollback \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Rolling back due to increased error rate"
  }'
```

**Via CLI:**

Trigger the rollback directly on the job:

```bash
# Rollback to previous version
expanso-cli job rollback my-job --reason "Manual rollback due to errors"

# Or rollback to a specific version
expanso-cli job rollback my-job --to-version 3 --reason "Rollback to known good version"
```

**Want to preview the rollback first?** Use dry run mode to see what would happen without actually triggering it:

```bash
curl -X POST https://cloud.expanso.io/api/v1/jobs/{job-id}/rollback \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dry_run": true
  }'
```

This returns what would happen—which version you'd roll back to, any warnings, and whether the rollback can proceed—without actually making changes.

### What Happens During Rollback

Once you trigger a rollback, here's what happens behind the scenes:

**First, the system validates that rollback is possible.** The deployment can't be in a terminal state like `completed` or `canceled`, and there must be a previous version to roll back to (you can't roll back your first deployment—there's nothing to roll back to).

**Your workspace atomically performs the rollback in a single transaction:**

1. **Updates the job spec** to the target version's configuration
2. **Cancels the active deployment** (if one exists) and marks it as failed
3. **Creates a new immediate deployment** with the rolled-back spec
4. **Increments the job version** to create a new version number

**The new deployment rolls out immediately** using the immediate deployment strategy. All nodes receive the rolled-back configuration at once, ensuring consistent state across your fleet.

**The job version increments** even though you're using an older spec. For example: v1 → v2 (fails) → v3 (with v1's spec). This preserves a clear audit trail showing the rollback occurred.

### Understanding the Rollback Response

The rollback API returns information about what happened:

```json
{
  "job_id": "job-abc123",
  "deployment_id": "deploy-new456",
  "from_version": 3,
  "to_version": 4,
  "target_spec": 2,
  "created_new": true,
  "cancelled_prior": true,
  "warnings": []
}
```

| Field | Description |
|-------|-------------|
| `job_id` | ID of the job that was rolled back |
| `deployment_id` | ID of the new rollback deployment that was created |
| `from_version` | Version you're rolling back from (e.g., 3) |
| `to_version` | New version number after rollback (e.g., 4) - always increments |
| `target_spec` | Version whose spec was used for rollback (e.g., 2) |
| `created_new` | Always true - rollback creates a new deployment |
| `cancelled_prior` | True if an active deployment was cancelled |
| `warnings` | Any non-fatal issues encountered during rollback |

### Rollback Limitations

Not all deployments can be rolled back. Here's what you need to know:

| Can Roll Back | Cannot Roll Back |
|---------------|------------------|
| Failed deployments | First deployment (no previous version exists) |
| In-progress deployments | Already completed deployments |
| Pending deployments | Already canceled deployments |

:::tip[Why Can't I Roll Back Completed Deployments?]
Once a deployment successfully completes, it's considered stable. To revert to an earlier version, create a new deployment targeting that version rather than rolling back the completed one.
:::

### Monitoring Rollback Progress

After triggering a rollback, you'll want to watch it progress. Here's how to monitor the state changes:

```bash
# Check the job rollout status
expanso-cli job describe my-job

# Check execution states
expanso-cli execution list --job-id my-job

# Verify executions at the rollback version are coming online
expanso-cli execution list --job-id my-job --job-version <to-version>
```

**What to look for:**

- **Job status**: Should show `rolling rollback (in progress)` in the ROLLOUT section
- **Rollback Version**: Indicates the target version for the rollback
- **Progress**: Shows how many nodes have been rolled back
- **Executions**: Should show the rollback version starting across all nodes

:::caution[Rollback Takes Time]
Depending on your fleet size and network conditions, rollback can take several minutes. Edge nodes need to stop current executions and start new executions with the rolled-back configuration. This is normal—monitor progress and ensure nodes are transitioning as expected.
:::

## Configuration Examples

### Basic Deployment with Health Checks

```yaml
name: data-processor
type: pipeline
config:
  input:
    http_server:
      path: /ingest
  pipeline:
    processors:
      - mapping: |
          root.processed = true
          root.timestamp = now()
  output:
    kafka:
      addresses: ["kafka:9092"]
      topic: processed-data

rollout:
  strategy: rolling
  health_check:
    interval: 10s
    success_threshold: 2
    failure_threshold: 3
    max_error_rate: 0.10
    deadline: 2m

selector:
  match_labels:
    env: production
    region: us-west
```

**Behavior:**
- Health is evaluated every 10 seconds (default)
- Needs 2 consecutive healthy windows to be validated (20 seconds minimum)
- 3 consecutive unhealthy windows trigger rollback (30 seconds of sustained errors)
- If not healthy within 2 minutes, rollback occurs
- Automatic rollback to previous version if validation fails

### Conservative Deployment for Critical Services

```yaml
name: critical-api
type: pipeline
config:
  input: { http_server: { path: /ingest } }
  output: { stdout: {} }

rollout:
  strategy: rolling
  health_check:
    interval: 30s              # Longer evaluation windows
    success_threshold: 4       # More consecutive healthy windows
    failure_threshold: 2       # Fail faster on errors
    max_error_rate: 0.05       # Stricter error rate (5%)
    deadline: 10m              # More time to stabilize

selector:
  match_labels:
    service: critical
    env: production
```

**Behavior:**
- Health evaluated every 30 seconds (longer windows for stability)
- Requires 4 consecutive healthy windows (2 minutes minimum)
- Only 2 unhealthy windows needed to trigger rollback (fail fast)
- Stricter error threshold of 5% instead of default 10%
- More conservative approach for critical infrastructure

### Fast-Paced Deployment for Non-Critical Services

```yaml
name: log-aggregator
type: pipeline
config:
  input: { http_server: { path: /ingest } }
  output: { stdout: {} }

rollout:
  strategy: rolling
  health_check:
    interval: 5s               # Faster evaluation
    success_threshold: 2       # Default threshold
    failure_threshold: 3       # Default threshold
    deadline: 1m               # Fast feedback

selector:
  match_labels:
    criticality: low
```

**Behavior:**
- Health evaluated every 5 seconds (faster feedback)
- Only needs 2 consecutive healthy windows (10 seconds minimum)
- Fails fast (1 minute deadline) for quick iteration
- Suitable for non-critical workloads where speed matters

## Monitoring Deployments

Once you've kicked off a job deployment, you'll want to track how it's progressing across your edge nodes. Expanso gives you several ways to monitor rollout status, check which nodes have completed validation, and identify any issues that come up during the deployment process.

### Check Job Status

To see the overall rollout status and progress for a job:

```bash
expanso-cli job describe data-processor
```

When a rollout is active, you'll see a ROLLOUT section with detailed progress information:

```
ROLLOUT
-------
Status:            rolling (in progress)
Progress:          75% (3/4 nodes)
Wave:              2/3
Failed Nodes:      0
Health Check:      5m deadline, 10% max error rate
Started:           2025-01-03T10:15:30Z
```

The ROLLOUT section shows you everything you need to know about the deployment. The **Status** field tells you which rollout strategy is running (rolling or immediate) and what state it's in—in progress, paused, failed, or completed. **Progress** shows the percentage complete along with how many nodes have been updated out of the total. For rolling deployments, **Wave** indicates which wave you're currently in and how many total waves there are. If any nodes fail their health checks, you'll see a **Failed Nodes** count. The **Health Check** line summarizes your validation configuration, and **Started** shows when the rollout began. Once the rollout finishes, you'll also see a **Completed** timestamp.

When a job is actively running a rollback, the Status field shows the rollback type and which version you're rolling back to:

```
Status:            rolling rollback (in progress)
Rollback Version:  42
```

If the rollout is paused or has failed, you'll also see the **Rollback Version** field indicating the target version for the rollback.

### Check Job List

To get a quick overview of all jobs and their rollout status:

```bash
expanso-cli job list
```

Jobs that are actively running a rollback operation show a `(rollback)` indicator in the VERSION column:

```
NAME              TYPE      STATE         VERSION          LABELS
data-processor    pipeline  deploying     43 (rollback)    env=prod
log-aggregator    pipeline  running       15               env=prod
```

You'll see the `(rollback)` indicator when the rollout type is "rollback" and the job state is `deploying`, `rollout_paused`, or `rollout_failed`.

### Check Execution State

To monitor execution states and see how validation is progressing on individual nodes:

```bash
# List executions for a job
expanso-cli execution list --job-id data-processor

# Sample output:
# EXECUTION ID       STATE        NODE                STARTED
# exec-abc-123      validating   edge-node-01        2m ago
# exec-xyz-789      running      edge-node-02        15m ago (stable)
```

The execution state tells you where each node is in the validation process. When you see `validating`, that execution is currently in its validation window. Once it passes validation, the state changes to `running` and gets marked as stable. If an execution hits `failed`, that means it failed validation and will trigger (or has triggered) a rollback.

### View Execution Details

For detailed information about a specific execution:

```bash
expanso-cli execution describe exec-abc-123
```

The output now includes rollout-specific fields:

```
EXECUTION DETAILS
-----------------
ID:              exec-abc-123
Job:             data-processor
Version:         43
State:           validating
Node:            edge-node-01
Created:         2025-01-03T10:20:45Z
Updated:         2025-01-03T10:21:15Z
Rollout Wave:    2
```

Once the execution passes validation and becomes stable, you'll see additional information:

```
EXECUTION DETAILS
-----------------
ID:              exec-xyz-789
Job:             data-processor
Version:         43
State:           running
Node:            edge-node-02
Created:         2025-01-03T10:15:30Z
Updated:         2025-01-03T10:16:45Z
Rollout Wave:    1
Stable At:       2025-01-03T10:16:45Z
```

The **Rollout Wave** field shows which wave this execution belongs to (only displayed if greater than 0). Once validation completes, you'll see **Stable At** with the timestamp when the execution passed its health checks. Use these fields to track validation progress and identify which executions have completed the health check window.

### View Health Diagnostics

When a deployment fails validation, detailed diagnostics help you pinpoint exactly what went wrong. Let's look at the health information available in execution status.

**Check execution health:**

```bash
expanso-cli execution describe exec-abc-123
```

When you describe an execution, you'll see these health diagnostic fields:

- `status`: Pending, Healthy, or Unhealthy
- `message`: Human-readable diagnostic showing component label and type (e.g., "Data Filter [bloblang]: error rate 15.2% exceeds threshold 10.0%")
- `details`: Machine-parseable metadata including:
  - `component_id`: UUID from visual builder (if set)
  - `component_label`: User-friendly component name (if set)
  - `component_name`: pipeline component type (e.g., "bloblang", "kafka", "http_client")
  - `component_type`: Component category (input, processor, or output)
  - `component_path`: component path within the pipeline (e.g., "root.pipeline.processors.0")
  - `failure_type`: Reason for failure (connection_failed, connection_lost, or error_rate)
  - `consecutive_unhealthy_windows`: How many consecutive windows were unhealthy
  - `error_count`: Number of errors in the failing component
  - `message_count`: Number of messages processed
  - `error_rate`: Calculated error rate percentage
  - `error_message`: Most recent error message from the component

You should see output like:

```
Health Status: Unhealthy
Message: Kafka Output [kafka]: connection failed 3 times (3 consecutive unhealthy windows)
Details:
  component_id: uuid-output-kafka-1
  component_label: Kafka Output
  component_name: kafka
  component_type: output
  component_path: root.output
  failure_type: connection_failed
  consecutive_unhealthy_windows: 3
  connection_failed_count: 3
  error_message: dial tcp 10.0.1.5:9092: connect: connection refused
```

:::tip[What These Diagnostics Tell You]
This level of detail lets you identify the exact component causing failures, understand whether it's a connection issue or processing error, see the actual error messages to diagnose root cause, and determine if the issue is transient or sustained.
:::

### Check for Rollback Events

```bash
# View execution history to see state transitions
expanso-cli execution history exec-abc-123

# Look for transitions like:
# Validating → Failed (triggers rollback)
# Then new deployment created with old spec
```

## Best Practices

### 1. Set Appropriate Thresholds

**success_threshold** (default: 2 windows):
- **Too low (1 window):** Single lucky success passes validation, more false positives
- **Too high (5+ windows):** Delays safe rollouts, increases deployment time
- **Recommendation:**
  - Use default (2) for most services
  - Use 3-4 for critical services requiring extra confidence
  - Use 1 for fast-iteration development environments

**failure_threshold** (default: 3 windows):
- **Too low (1-2 windows):** Transient errors cause unnecessary rollbacks
- **Too high (5+ windows):** Bad deployments stay active longer before rollback
- **Recommendation:**
  - Use default (3) for most services
  - Use 2 for critical services (fail faster)
  - Use 4-5 for non-critical services (more tolerance)

**interval** (default: 10s):
- **Too short (< 5s):** Noisy evaluation, not enough data per window
- **Too long (> 30s):** Slower feedback, delays detection of issues
- **Recommendation:**
  - Use default (10s) for most services
  - Use 5s for fast-feedback development
  - Use 20-30s for services with variable traffic patterns

### 2. Set Realistic Deadlines

Deadline should account for:
- Minimum validation time: `success_threshold * interval` (e.g., 2 × 10s = 20s)
- Startup time (loading configs, establishing connections)
- Initialization delays (cache warming, etc.)
- Traffic ramp-up time (if no traffic = no health evaluation)

**Formula:**
```
Deadline = (success_threshold × interval) + StartupTime + Buffer
```

**Example:**
```
success_threshold: 3
interval: 10s
startup_time: 15s
buffer: 30s

Deadline = (3 × 10s) + 15s + 30s = 75s ≈ 2m
```

### 3. Test Rollback Scenarios

Verify rollback works before production:

```bash
# Deploy a job version that will fail
expanso-cli job deploy broken-version.yaml

# Check the rollout status, which shows the deployment and rollback state
expanso-cli job describe my-job

# Verify job version incremented with old spec
expanso-cli job versions my-job
```

### 4. Monitor Deployment Metrics

Track these metrics:
- Validation success rate
- Time spent in validation state
- Rollback frequency
- Time to rollback

### 5. Handle First Deployments

For the first deployment of a job:
- No rollback target exists
- Failed validation uses normal retry logic
- Consider more conservative health check settings

## Troubleshooting

### Execution Stuck in Validating State

**Symptoms:** Execution remains in `validating` state for extended period

**Diagnosis:**
```bash
expanso-cli execution describe <exec-id>
# Check: created_at vs current time
# Check: health_check deadline configuration
```

**Possible Causes:**
1. Execution is flapping (alternating healthy/unhealthy windows)
2. Not receiving traffic (idle windows don't count toward success threshold)
3. Deadline is very long
4. Success threshold requirements not yet met

**Solution:**
- Check edge node logs for execution errors
- Verify execution is actually processing data (needs traffic for health evaluation)
- Look for alternating healthy/unhealthy patterns (indicates instability)
- Consider adjusting thresholds if too conservative

### Rollback Not Occurring

**Symptoms:** Execution fails but doesn't trigger rollback

**Diagnosis:**
```bash
expanso-cli execution describe <failed-exec-id>
# Check: rollback_to_version field (should be > 0)
# Check: stable_at field (should be empty if new deployment)

# Check the job's rollout state
expanso-cli job describe <job-name>
```

**Possible Causes:**
1. First deployment (no previous version to roll back to)
2. Execution was previously stable (has `stable_at` set)
3. Job doesn't have deployment configuration
4. Your workspace hasn't triggered rollback yet (waiting for threshold)

**Solution:**
- Verify deployment configuration exists in job spec
- Check if previous version exists using `expanso-cli job versions <job>`
- Review execution history to see if it was previously validated
- For cluster-wide issues, check if your workspace has a rollback in progress

### Manual Rollback Fails

**Symptoms:** Manual rollback API call returns error

**Diagnosis:**
```bash
# Try with dry_run to see what would happen
curl -X POST https://cloud.expanso.io/api/v1/jobs/{job-id}/rollback \
  -H "Authorization: Bearer $EXPANSO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dry_run": true}'
```

**Common Errors:**

**"rollback already in progress"**
- Another rollback is in progress
- Wait for current rollback to complete
- Check job state: `expanso-cli job describe <job-id>`

**"rollout in terminal state"**
- Rollout already completed or canceled
- Cannot roll back completed rollouts
- If you need to revert, deploy the previous version as a new deployment

**"no previous version to rollback to"**
- This is the first deployment (RollbackToVersion = 0)
- No previous version exists
- Stop the job and redeploy with fixes: `expanso-cli job stop <job-id>`

**Solution:**
- Review error message for specific constraint violated
- Check deployment state and history
- Use dry run mode to validate before attempting rollback

### First Deployment Fails Validation

**Symptoms:** First deployment of a job fails validation and execution stops (doesn't retry)

**Diagnosis:**
```bash
# Check if this is first deployment
expanso-cli job describe my-job
# If this is the first deployment, there's no previous version to roll back to

# Check execution state
expanso-cli execution describe <exec-id>
# Should show: state=stopped, message about no rollback target
```

**Behavior:**
- First deployment has no previous version to roll back to
- Edge nodes stop failed executions instead of retrying indefinitely
- This prevents infinite retry loops on permanently broken deployments

**Solution:**
1. Fix the underlying issue (check validation failure cause)
2. Deploy corrected version:
   ```bash
   # Update job configuration
   expanso-cli job deploy fixed-version.yaml
   ```
3. Consider more lenient health check thresholds for initial deployment:
   ```yaml
   rollout:
     strategy: rolling
     health_check:
       failure_threshold: 5  # More tolerance for startup issues
       deadline: 10m         # More time to stabilize
   ```

### Infinite Rollback Loops

**Symptoms:** Job keeps switching between versions

**This should not happen** due to StableAt protection, but if it does:

**Diagnosis:**
```bash
# Check execution history
expanso-cli execution history <exec-id-version-a>
expanso-cli execution history <exec-id-version-b>

# Look for stable_at timestamps
expanso-cli execution describe <exec-id>
```

**Solution:**
- Report as a bug - StableAt mechanism should prevent this
- Manually stop problematic job: `expanso-cli job stop <job-name>`
- Deploy a known-good version

### Target Version Missing from History

**Symptoms:** Rollback fails with "target version not found" error

**Diagnosis:**
```bash
# Check what versions exist in history
expanso-cli job versions <job-name>

# Check if the target version spec exists
expanso-cli job describe <job-name> --version <target-version>
```

**Possible Causes:**
1. Target version was never created (version gap in history)
2. Job version history is incomplete or corrupted
3. Requested version number doesn't exist

**Solution:**
- Verify the version exists in history: `expanso-cli job versions <job>`
- Check for gaps in version sequence
- If version history is corrupt, manually deploy a known-good configuration
- Use `--version` flag to specify a version that exists

## When to Use Deployments

**Use deployment configuration when:**
- Updating production services that require high availability
- Deploying to critical edge infrastructure
- You want automatic safety nets for bad deployments
- You're rolling out changes across a large edge fleet

**Skip deployment configuration when:**
- Rapidly iterating in development environments
- Deploying one-off data processing jobs
- Managing system/operational jobs (cleanup tasks, etc.)
- Immediate updates are acceptable (no validation period needed)

## Related Documentation

- [Edge Node Deployment](/operations/deployment/edge-deployment): How to deploy and manage edge nodes themselves
- [Fleet Monitoring](/operations/monitoring/fleet-monitoring): Monitor deployment health across your fleet
- [CLI Job Commands](/references/cli/expanso-cli/job/): Manage job deployments via CLI
- [Execution States](/references/cli/expanso-cli/execution/): Understanding execution lifecycle

## Monitoring

# Monitoring

Monitor the health and performance of your Expanso Edge nodes and pipelines.

There are two ways to monitor your pipelines:

- **Monitoring tab in the Expanso Cloud console (primary).** Built-in graphs for each pipeline, with no setup required. This is the fastest way to see throughput, errors, and resource usage.
- **OpenTelemetry export (advanced).** Export edge node metrics over OTLP to your own observability stack (Prometheus, Grafana, Datadog), where you run your own queries, dashboards, and alerts. See [OpenTelemetry Metrics Export](/operations/monitoring/opentelemetry).

## Metrics

Track pipeline performance and resource utilization:

- Throughput and latency
- Error rates
- Resource usage (CPU, memory)

The console Monitoring tab shows these per pipeline. For the full set of exported metrics and their names, see [OpenTelemetry Metrics Export](/operations/monitoring/opentelemetry).

## Logging

Access detailed logs from your edge nodes:

- Structured logging (JSON) with configurable log levels
- Local logs via journald or a log file (see [Systemd Service](/operations/deployment/headless-systemd))
- Ship logs to your own aggregator for centralized search and analysis

## Alerting

Alerting is bring-your-own. Export metrics over OTLP and define alert rules in your own Prometheus or Grafana. You can alert on the same edge metrics you export, for example error rates or pipeline readiness.

## What's Next?

- **[OpenTelemetry Metrics Export](/operations/monitoring/opentelemetry)**: Export edge metrics to your observability stack
- **[Edge Node Configuration](/operations/configuration)**: Configure logging and telemetry

## Monitor Your Edge Fleet

# Monitor Your Edge Fleet

:::info[Private orchestrator deployments]
This guide covers monitoring for orchestrators running in your own environment. That option is available on request: contact the Expanso team at [support@expanso.io](mailto:support@expanso.io) for details and to get set up. If you use the standard **Managed Expanso Cloud** service, monitor pipelines from the **Monitoring** tab in the console instead.
:::

You've deployed jobs to edge nodes, and they're processing data across your distributed infrastructure. But how do you know everything's working? When a node goes offline, when a job fails, or when throughput drops—you need to know immediately.

In this tutorial, you'll build a complete monitoring stack for Expanso using Prometheus and Grafana. You'll set up metrics collection, create dashboards to visualize fleet health, configure alerts for critical issues, and learn how to debug problems using observability data. By the end, you'll have production-grade monitoring that keeps you informed about your edge infrastructure.

This tutorial takes about 45-60 minutes to complete.

## What You'll Learn

- How to configure Expanso to export metrics via OpenTelemetry
- How to set up Prometheus to collect metrics from orchestrator and edge nodes
- How to create Grafana dashboards for job and node health visualization
- How to monitor job execution states and throughput
- How to track node connectivity, heartbeats, and resources
- How to configure alerts for node disconnections, job failures, and performance issues
- How to correlate logs with metrics for debugging
- How to identify and troubleshoot common edge infrastructure problems

## Prerequisites

Before starting, make sure you have:

- **A private orchestrator deployment arranged with Expanso** (contact [support@expanso.io](mailto:support@expanso.io) to get set up) and the deployment package supplied for your environment. This guide is not an installation path.
- **A running Expanso deployment** with orchestrator and at least one edge node (from the [First Edge Deployment](/operations/deployment/edge-deployment) tutorial)
- **At least one deployed job** actively processing data
- **Docker or Podman** installed for running Prometheus and Grafana
- **Basic familiarity** with Prometheus query language (PromQL) is helpful but not required

:::tip[Production vs. Development]
This tutorial uses Docker Compose for quick setup. For production deployments, you'll want to run Prometheus and Grafana on dedicated infrastructure with persistent storage, high availability, and proper security configurations.
:::

## Step 1: Understand Expanso's Metrics Architecture

Before diving into configuration, let's understand what metrics Expanso exposes and how they flow through the observability stack.

**Expanso uses OpenTelemetry for metrics export**, which means it can send data to any OTLP-compatible collector. The architecture looks like this:

```mermaid
graph LR
    Edge[Edge Nodes] -->|OTLP| Collector[OTLP Collector]
    Orch[Orchestrator] -->|OTLP| Collector
    Collector -->|Prometheus Remote Write| Prom[Prometheus]
    Prom --> Grafana[Grafana]
    Grafana -->|Queries| Prom
```

**Key metric categories:**

1. **Process metrics** (CPU, memory, file descriptors) - Always enabled
2. **Go runtime metrics** (GC, goroutines, heap) - Optional, disabled by default
3. **Job execution metrics** (state, throughput, errors) - Application-specific
4. **Node connection metrics** (heartbeats, session state) - Network health
5. **Deployment metrics** (rollout progress, health checks) - Job lifecycle
6. **Disk metrics** (available space, directory sizes) - Always enabled

For this tutorial, we'll use the OpenTelemetry Collector in Prometheus mode, which directly scrapes metrics and stores them.

## Step 2: Configure Orchestrator to Export Metrics

Let's configure the Expanso orchestrator to export metrics via OpenTelemetry.

**Update your orchestrator configuration:**

Add the telemetry section to `/etc/expanso/orchestrator-config.yaml`. Download the complete configuration:

```bash
curl -O https://docs.expanso.io/examples/monitoring/orchestrator-telemetry-config.yaml
```

Or [view the configuration file](pathname:///examples/monitoring/orchestrator-telemetry-config.yaml)

Key settings:
- `endpoint`: OTLP collector address (localhost:4317 for co-located collector)
- `export_interval`: How often to export metrics (15s recommended)
- `include_go_metrics`: Enable detailed Go runtime metrics
- `resource_attributes`: Add custom labels for filtering in Prometheus

**Restart the orchestrator to apply the configuration:**

```bash
sudo systemctl restart expanso-orchestrator

# Verify telemetry is initialized
sudo journalctl -u expanso-orchestrator -n 50 | grep -i telemetry
```

You should see log entries indicating telemetry initialization:

```
INFO Telemetry enabled: endpoint=localhost:4317 protocol=grpc
INFO Process metrics collector initialized: interval=15s
INFO Go metrics collector initialized: interval=15s
```

:::caution[Connection Errors Are Normal]
Until we start the OTLP collector in the next step, you'll see connection errors in the logs. This is expected—the orchestrator will retry connecting automatically.
:::

## Step 3: Configure Edge Nodes to Export Metrics

Now let's configure edge nodes to send metrics to the same collector.

**Update each edge node's configuration:**

Add the telemetry section to `/etc/expanso/edge-config.yaml`. Download the configuration:

```bash
curl -O https://docs.expanso.io/examples/monitoring/edge-telemetry-config.yaml
```

Or [view the configuration file](pathname:///examples/monitoring/edge-telemetry-config.yaml)

**Important:** Update the `endpoint` to point to your OTLP collector (typically the orchestrator hostname).

**Restart edge nodes:**

```bash
# On each edge node
sudo systemctl restart expanso-edge

# Verify telemetry configuration
sudo journalctl -u expanso-edge -n 20 | grep -i telemetry
```

:::tip[Centralized vs. Distributed Collection]
In this tutorial, edge nodes send metrics directly to a central OTLP collector. For large-scale deployments, you can run a local collector on each edge site and aggregate to a central Prometheus instance. This reduces network traffic and improves resilience.
:::

## Step 4: Set Up the OpenTelemetry Collector and Prometheus

Now let's deploy the observability stack using Docker Compose.

**Create a directory for monitoring configuration:**

```bash
mkdir -p ~/expanso-monitoring/{prometheus,grafana,otel-collector}
cd ~/expanso-monitoring
```

**Create the OpenTelemetry Collector configuration:**

Download the configuration file:

```bash
curl -o otel-collector/config.yaml https://docs.expanso.io/examples/monitoring/otel-collector-config.yaml
```

Or [view the configuration file](pathname:///examples/monitoring/otel-collector-config.yaml)

**Create the Prometheus configuration:**

Download the configuration file:

```bash
curl -o prometheus/prometheus.yml https://docs.expanso.io/examples/monitoring/prometheus.yml
```

Or [view the configuration file](pathname:///examples/monitoring/prometheus.yml)

**Create the Docker Compose file:**

Download the complete stack configuration:

```bash
curl -o docker-compose.yml https://docs.expanso.io/examples/monitoring/docker-compose.yml
```

Or [view the configuration file](pathname:///examples/monitoring/docker-compose.yml)

This sets up:
- **OTLP Collector** on ports 4317 (gRPC) and 4318 (HTTP)
- **Prometheus** on port 9091 (web UI)
- **Grafana** on port 3000

**Start the monitoring stack:**

```bash
docker compose up -d

# Verify all containers are running
docker compose ps
```

You should see all three containers running:

```
NAME                      STATUS          PORTS
expanso-otel-collector    Up 10 seconds   0.0.0.0:4317-4318->4317-4318/tcp
expanso-prometheus        Up 10 seconds   0.0.0.0:9091->9090/tcp
expanso-grafana           Up 10 seconds   0.0.0.0:3000->3000/tcp
```

**Verify metrics are flowing:**

```bash
# Check OTLP collector logs
docker compose logs -f otel-collector

# You should see messages about receiving metrics
```

:::tip[Firewall Configuration]
If your orchestrator and edge nodes are on different machines, make sure port 4317 (OTLP gRPC) is open on the collector host. Use `sudo ufw allow 4317/tcp` or equivalent for your firewall.
:::

## Step 5: Verify Metrics in Prometheus

Let's verify that Prometheus is collecting metrics from Expanso components.

**Open the Prometheus web UI:**

Navigate to `http://localhost:9091` in your browser.

**Run a basic query to verify data collection:**

In the query box, enter:

```promql
process_cpu_utilization_ratio
```

Click "Execute" and switch to the "Graph" tab. You should see CPU utilization metrics for both the orchestrator and any connected edge nodes.

**Check available metrics:**

Click the "Graph" dropdown and explore the available metrics. You should see:

```
Process Metrics:
  - process_cpu_utilization_ratio
  - process_memory_usage_bytes
  - process_memory_virtual_bytes
  - process_open_file_descriptors_ratio
  - process_max_file_descriptors_ratio

Go Runtime Metrics (if enabled):
  - go_memory_heap_alloc_bytes
  - go_memory_heap_sys_bytes
  - go_gc_cycles_total
  - go_gc_pause_last_seconds
  - go_goroutines_count_ratio
  - go_max_procs

Log Streaming Metrics (orchestrator only):
  - websocket_proxy_connections_active
  - websocket_proxy_connections_total
  - websocket_proxy_messages_client_to_server_total
  - websocket_proxy_messages_server_to_client_total
  - websocket_proxy_disconnections_total

NATS Transport Metrics:
  - nats_connection_status (1=connected, 0=disconnected) - available on both orchestrators and edge nodes

Disk Metrics:
  - disk_available (available free space on data directory filesystem)
  - disk_state_size (total size of the state directory)
  - disk_executions_size (total size of the executions directory, edge only)
```

**Query metrics by service:**

To see only orchestrator metrics:

```promql
process_memory_usage_bytes{service_name="expanso-orchestrator"}
```

To see only edge node metrics:

```promql
process_memory_usage_bytes{service_name="expanso-edge"}
```

**Check metric cardinality:**

Count how many unique service instances are reporting:

```promql
count by (service_name, node_id) (process_cpu_utilization_ratio)
```

You should see one entry for the orchestrator and one for each connected edge node.

:::tip[No metrics appearing?]
See the [Metrics Not Appearing in Prometheus](#metrics-not-appearing-in-prometheus) troubleshooting section below.
:::

## Step 6: Create Grafana Dashboards for Fleet Monitoring

Now let's create comprehensive dashboards to visualize your edge fleet's health.

**Log into Grafana:**

Navigate to `http://localhost:3000` and log in with:
- Username: `admin`
- Password: `admin`

You'll be prompted to change the password on first login.

**Add Prometheus as a data source:**

1. Click the gear icon (⚙️) in the left sidebar → "Data sources"
2. Click "Add data source"
3. Select "Prometheus"
4. Configure:
   - Name: `Expanso Prometheus`
   - URL: `http://prometheus:9090`
   - Access: `Server (default)`
5. Click "Save & Test"

You should see a success message: "Successfully queried the Prometheus API."

**Create the Fleet Overview Dashboard:**

1. Click the "+" icon → "Create Dashboard"
2. Click "Add visualization"
3. Select "Expanso Prometheus" as the data source

Let's create several panels:

**Panel 1: Node Connection Status**

This panel shows which nodes are connected and healthy.

```promql
# Query
count by (node_id, node_hostname) (
  process_cpu_utilization_ratio{service_name="expanso-edge"}
)
```

- Panel type: **Stat**
- Title: **Connected Edge Nodes**
- Description: "Number of edge nodes currently reporting metrics"
- Value options: **Last (not null)**
- Color scheme: **Green-Yellow-Red (by value)**
- Thresholds:
  - Green: > 0
  - Red: 0

**Panel 2: CPU Utilization by Node**

Shows CPU usage across all edge nodes and orchestrator.

```promql
# Query
process_cpu_utilization_ratio * 100
```

- Panel type: **Time series**
- Title: **CPU Utilization (%)**
- Legend: `{{service_name}} - {{node_hostname}}`
- Unit: **Percent (0-100)**
- Y-axis min: 0
- Y-axis max: 100

**Panel 3: Memory Usage by Node**

Tracks memory consumption across the fleet.

```promql
# Query (convert to MB)
process_memory_usage_bytes / 1024 / 1024
```

- Panel type: **Time series**
- Title: **Memory Usage (MB)**
- Legend: `{{service_name}} - {{node_hostname}}`
- Unit: **megabytes**

**Panel 4: Goroutine Count (Go Runtime)**

Monitor goroutine growth to detect leaks.

```promql
# Query
go_goroutines_count_ratio
```

- Panel type: **Time series**
- Title: **Goroutine Count**
- Legend: `{{service_name}}`
- Y-axis min: 0

**Panel 5: Garbage Collection Frequency**

Track GC activity across services.

```promql
# Query: GC cycles per minute
rate(go_gc_cycles_total[5m]) * 60
```

- Panel type: **Time series**
- Title: **GC Cycles per Minute**
- Legend: `{{service_name}}`
- Unit: **ops/min**

**Panel 6: File Descriptor Usage**

Monitor file descriptor utilization to prevent exhaustion.

```promql
# Query: FD usage percentage
(process_open_file_descriptors_ratio /
 process_max_file_descriptors_ratio) * 100
```

- Panel type: **Gauge**
- Title: **File Descriptor Usage (%)**
- Legend: `{{service_name}}`
- Unit: **Percent (0-100)**
- Thresholds:
  - Green: < 50
  - Yellow: 50-80
  - Red: > 80

**Save the dashboard:**

Click the save icon (💾) at the top right, name it "Expanso Fleet Overview", and click "Save".

:::tip[Dashboard Variables]
For production dashboards, add variables to filter by environment, region, or specific nodes. In dashboard settings, add variables like `$environment` or `$node_id` to make dashboards interactive.
:::

## Step 7: Monitor Job Execution Health

Let's create a dashboard focused on job execution state and throughput.

**Create a new dashboard:** Click "+" → "Create Dashboard"

**Panel 1: Jobs by Execution State**

This requires metrics from the orchestrator's job controller. While Expanso doesn't expose execution state as Prometheus metrics by default (it's accessible via the API), we can infer health from other signals.

For now, let's monitor job execution indirectly through resource usage patterns.

**Panel 2: Edge Node Heartbeat Latency**

Monitor the time between heartbeats to detect connectivity issues.

```promql
# Query: Time since last heartbeat (approximation)
time() - timestamp(process_cpu_utilization_ratio{service_name="expanso-edge"})
```

- Panel type: **Time series**
- Title: **Heartbeat Freshness (seconds)**
- Legend: `{{node_hostname}}`
- Unit: **seconds**
- Alert threshold: > 90 seconds (Disconnected state)

**Panel 3: Process Memory Growth Rate**

Detect memory leaks by tracking growth rate.

```promql
# Query: Memory growth in MB per hour
rate(process_memory_usage_bytes[1h]) * 3600 / 1024 / 1024
```

- Panel type: **Time series**
- Title: **Memory Growth Rate (MB/hour)**
- Legend: `{{service_name}} - {{node_hostname}}`
- Unit: **MB/h**

**Panel 4: GC Pause Times**

Monitor garbage collection impact on performance.

```promql
# Query: Last GC pause in milliseconds
go_gc_pause_last_seconds * 1000
```

- Panel type: **Time series**
- Title: **GC Pause Time (ms)**
- Legend: `{{service_name}}`
- Unit: **milliseconds**
- Alert threshold: > 100ms

**Panel 5: CPU Utilization Heatmap**

Visualize CPU patterns across all nodes.

```promql
# Query
process_cpu_utilization_ratio * 100
```

- Panel type: **Heatmap**
- Title: **CPU Utilization Heatmap**
- Legend: `{{node_hostname}}`
- Color scheme: **Green-Yellow-Red**

**Save this dashboard** as "Expanso Job & Node Health".

## Step 8: Set Up Alerts for Critical Issues

Let's configure Prometheus alerting rules for common edge infrastructure problems.

**Create the alerts directory:**

```bash
mkdir -p ~/expanso-monitoring/prometheus/alerts
```

**Create alert rules for node health:**

Download the alert rules:

```bash
mkdir -p prometheus/alerts
curl -o prometheus/alerts/node-health.yml https://docs.expanso.io/examples/monitoring/node-health-alerts.yml
```

Or [view the alert rules](pathname:///examples/monitoring/node-health-alerts.yml)

This includes alerts for:
- Edge node disconnections (>2 minutes of missed metrics)
- Orchestrator downtime
- High CPU usage (>85% for 5 minutes)
- High memory usage (>1GB for 5 minutes)

**Create alert rules for resource exhaustion:**

Download the resource limit alerts:

```bash
curl -o prometheus/alerts/resource-limits.yml https://docs.expanso.io/examples/monitoring/resource-limits-alerts.yml
```

Or [view the alert rules](pathname:///examples/monitoring/resource-limits-alerts.yml)

This includes alerts for:
- High file descriptor usage (>80%)
- Memory leak detection (>10MB/s growth)
- Goroutine leaks (>5000 goroutines)
- Excessive GC activity (>2 cycles/second)

**Reload Prometheus configuration:**

```bash
# Reload config without restarting
curl -X POST http://localhost:9091/-/reload
```

**Verify alerts are loaded:**

Navigate to `http://localhost:9091/alerts` in Prometheus UI. You should see all configured alerts listed.

**Check alert states:**

The alerts will show as:
- **Green (Inactive)**: No issues detected
- **Yellow (Pending)**: Issue detected but waiting for `for` duration
- **Red (Firing)**: Alert is active

## Step 9: Access and Correlate Logs

Metrics tell you when something's wrong, but logs tell you why. Let's integrate log access into your monitoring workflow.

**Accessing edge node logs:**

```bash
# On the edge node machine
sudo journalctl -u expanso-edge -n 100

# Follow in real-time
sudo journalctl -u expanso-edge -f

# Filter by time range
sudo journalctl -u expanso-edge --since "10 minutes ago"

# Export to file for analysis
sudo journalctl -u expanso-edge --since "1 hour ago" > edge-debug.log
```

**Accessing job logs:**

```bash
# List a job's executions
expanso-cli job executions syslog-processor

# Stream a job's logs live (optionally from a specific node)
expanso-cli job logs syslog-processor --node node-a1b2c3d4e5f6
```

**Correlating metrics with logs:**

When you see an alert fire in Grafana, here's how to debug:

1. **Note the timestamp** from the Grafana panel
2. **Identify the affected node** from the metric labels
3. **Query logs around that time:**

```bash
# If an edge node disconnected around 15:30, check its logs on the node
ssh user@edge-node
sudo journalctl -u expanso-edge --since "15:25" --until "15:35"
```

**Common log patterns to look for:**

```bash
# Node connection issues
grep "connection refused\|timeout\|network error" edge-debug.log

# Job execution failures
grep "execution failed\|pipeline error\|fatal" edge-debug.log

# Resource exhaustion
grep "out of memory\|too many open files\|disk full" *.log

# Authentication problems
grep "unauthorized\|authentication failed\|invalid token" *.log
```

## Stream Job Logs

Metrics tell you what is happening; logs tell you why. Expanso streams a running pipeline's logs live, so you can watch them as they happen or pick up from an earlier point.

### From the Expanso Cloud console

The primary way to view logs is the Expanso Cloud console. Open your pipeline and stream its logs live from the console. When a pipeline runs on more than one edge node, you can choose which node to stream from.

### From the CLI

You can stream the same logs from the command line:

```bash
# Stream a job's logs live, optionally from a specific node
expanso-cli job logs my-job --node node-123
```

### Directly from the API

The console and the CLI both use the same endpoint: a GET request that upgrades to a WebSocket.

```
GET /api/v1/jobs/{job-id}/logs
```

All query parameters are optional:

| Parameter | Description |
|-----------|-------------|
| `node_id` | Stream from a specific edge node. If omitted, the orchestrator selects one active execution of the job. |
| `start` | A timestamp in nanoseconds since the Unix epoch to begin streaming from. Defaults to the current moment (live tailing); set it to an earlier time to begin from that point. |

There is no `end` parameter and no separate log-level parameter. Closing the WebSocket stops the stream.

## Step 10: Debug Common Issues Using Metrics

Let's walk through real debugging scenarios using the monitoring stack you've built.

**Scenario 1: Edge Node Shows Intermittent Disconnections**

**Symptom:** Grafana shows gaps in metrics for a specific edge node.

**Investigation:**

1. Check the heartbeat freshness panel—you see periodic spikes to 120+ seconds
2. Query Prometheus for network quality:

```promql
# Gaps in metrics collection
resets(process_cpu_utilization_ratio{node_id="node-abc123"}[1h])
```

3. SSH to the edge node and check network:

```bash
# Ping orchestrator
ping -c 10 orchestrator.example.com

# Check packet loss
mtr orchestrator.example.com

# Review network errors
sudo journalctl -u expanso-edge -n 200 | grep -i "connection\|network"
```

**Common causes:**
- Unstable WiFi/cellular connection (increase heartbeat interval)
- Firewall intermittently blocking port 9090
- Network congestion during peak hours

**Solution:** Increase heartbeat tolerance or improve network infrastructure.

**Scenario 2: Orchestrator Memory Usage Growing Continuously**

**Symptom:** Memory growth rate panel shows consistent positive trend.

**Investigation:**

1. Check Go heap metrics in Grafana:

```promql
# Heap allocation over time
go_memory_heap_alloc_bytes{service_name="expanso-orchestrator"}

# Memory growth rate
rate(go_memory_heap_alloc_bytes[1h])
```

2. Compare heap vs. system memory:

```promql
# If heap is stable but RSS grows, it's a different issue
process_memory_usage_bytes{service_name="expanso-orchestrator"} -
go_memory_heap_alloc_bytes{service_name="expanso-orchestrator"}
```

3. Check goroutine count for leaks:

```promql
go_goroutines_count_ratio{service_name="expanso-orchestrator"}
```

4. If goroutines are growing, get a goroutine dump:

```bash
# Enable pprof endpoint (if configured)
curl http://localhost:6060/debug/pprof/goroutine?debug=1 > goroutines.txt

# Analyze the dump for stuck goroutines
grep -A 5 "created by" goroutines.txt | sort | uniq -c | sort -rn
```

**Common causes:**
- Goroutine leaks from unclosed connections
- Cached data not being evicted
- Large number of historical records in state store

**Solution:** Review the goroutine dump, fix leaks, or implement cache size limits.

**Scenario 3: Job Execution Not Processing Data**

**Symptom:** Job is marked as "running" but no output is produced.

**Investigation:**

1. Check if the edge node is connected:

```bash
expanso-cli node list
```

2. Get execution details:

```bash
expanso-cli execution describe exec-xyz789
```

3. Stream the job's logs from the affected node:

```bash
expanso-cli job logs <job-name> --node node-abc123
```

4. Look for resource constraints on the edge node:

```promql
# CPU at limits?
process_cpu_utilization_ratio{node_id="node-abc123"} > 0.95

# Memory at limits?
process_memory_usage_bytes{node_id="node-abc123"}
```

5. SSH to edge node and check local pipeline status:

```bash
# Check if the edge agent is running
sudo systemctl status expanso-edge

# Check local buffer for backed-up data
ls -lh /var/lib/expanso/buffer/
```

**Common causes:**
- Input source (file, network) is not producing data
- Pipeline filter is dropping all messages
- Output destination is unreachable (network partition)
- Resource limits preventing processing

**Solution:** Review pipeline configuration, check input/output connectivity, verify resource allocation.

## Step 11: Track Deployment Progress

When you roll out job updates, monitoring deployment progress prevents issues from affecting your entire fleet.

**Monitor rolling deployment progress:**

```bash
# Start a rolling deployment
expanso-cli job deploy updated-job.yaml

# Watch deployment status
watch -n 2 'expanso-cli job executions updated-job'
```

**In Grafana, create a deployment tracking panel:**

```promql
# Count nodes by execution version
count by (version) (
  # This requires custom metrics from the orchestrator
  # For now, monitor via API or CLI
)
```

**Best practices for deployment monitoring:**

1. **Watch for failures during rollout:**

```bash
# Monitor execution health during deployment
watch -n 5 'expanso-cli job executions my-job | grep -E "failed|error"'
```

2. **Verify new version before proceeding:**

```bash
# Check health of new version executions
expanso-cli job executions my-job --job-version 2 | grep healthy
```

3. **Roll back if issues detected:**

```bash
# If new version shows problems, roll back to previous version
expanso-cli job rollback my-job
```

**Create a deployment dashboard panel:**

Add to your Grafana dashboard to show deployment activity:

```promql
# Recent configuration changes (approximation via process restarts)
changes(process_cpu_utilization_ratio[5m])
```

## Step 12: Create Custom Alerts for Your Workload

Generic alerts are a start, but you'll want workload-specific alerts based on your edge computing use case.

**Example: Log Processing Pipeline Alerts**

If you're processing syslog data:

```yaml title="prometheus/alerts/syslog-pipeline.yml"
groups:
  - name: syslog_pipeline
    interval: 30s
    rules:
      # Alert if log processing throughput drops
      - alert: LowLogThroughput
        expr: |
          # This requires custom metrics from your pipeline
          # Placeholder: monitor CPU as proxy for activity
          avg(rate(process_cpu_utilization_ratio{
            service_name="expanso-edge"
          }[5m])) < 0.05
        for: 10m
        labels:
          severity: warning
          workload: syslog
        annotations:
          summary: "Log processing throughput is low"
          description: "Average edge node CPU usage is below 5%, indicating low log volume or processing issues"
```

**Example: IoT Data Collection Alerts**

If you're collecting sensor data:

```yaml title="prometheus/alerts/iot-collection.yml"
groups:
  - name: iot_data_collection
    interval: 30s
    rules:
      # Alert if edge nodes stop collecting data
      - alert: DataCollectionStalled
        expr: |
          # Monitor memory growth as proxy for data buffering
          rate(process_memory_usage_bytes{
            service_name="expanso-edge"
          }[10m]) == 0
        for: 15m
        labels:
          severity: warning
          workload: iot
        annotations:
          summary: "Data collection may have stalled on {{ $labels.node_hostname }}"
          description: "Edge node memory is not growing, which may indicate no data is being buffered"
```

**Reload Prometheus to apply custom alerts:**

```bash
curl -X POST http://localhost:9091/-/reload
```

## Verification Checklist

Let's verify your complete monitoring setup:

- ✅ Orchestrator configured to export metrics via OTLP
- ✅ Edge nodes configured to export metrics via OTLP
- ✅ OpenTelemetry Collector receiving metrics from all components
- ✅ Prometheus scraping metrics from OTLP Collector
- ✅ Grafana connected to Prometheus data source
- ✅ Fleet Overview dashboard created with key health metrics
- ✅ Job & Node Health dashboard created
- ✅ Alert rules configured for node disconnections
- ✅ Alert rules configured for resource exhaustion
- ✅ Alerts visible in Prometheus UI
- ✅ Log access configured for orchestrator and edge nodes
- ✅ Able to correlate metrics with logs for debugging
- ✅ Tested at least one debugging scenario
- ✅ Custom workload-specific alerts configured

If all items are checked, congratulations! You have production-grade monitoring for your edge fleet.

## What You Learned

You've built a comprehensive monitoring stack for Expanso:

- ✅ Configured OpenTelemetry metrics export from orchestrator and edge nodes
- ✅ Deployed and configured Prometheus for metrics collection
- ✅ Created Grafana dashboards for fleet health visualization
- ✅ Set up process metrics (CPU, memory, file descriptors)
- ✅ Enabled Go runtime metrics (GC, goroutines, heap)
- ✅ Configured alerts for critical issues (node disconnections, resource exhaustion)
- ✅ Integrated log access for debugging
- ✅ Practiced debugging common edge infrastructure problems
- ✅ Monitored deployment progress and health
- ✅ Created custom workload-specific alerts

## Key Concepts

**OpenTelemetry (OTLP):** Industry-standard protocol for exporting telemetry data (metrics, traces, logs). Expanso uses OTLP to send metrics to collectors, making it compatible with any OTLP-compatible backend.

**Process Metrics:** OS-level metrics about running processes including CPU utilization, memory usage (RSS and virtual), and file descriptor counts. Always enabled in Expanso.

**Go Runtime Metrics:** Go-specific metrics about garbage collection, goroutine counts, heap allocation, and memory management. Optional in Expanso, enabled via `include_go_metrics: true`.

**Heartbeats:** Periodic health reports from edge nodes to your workspace (every 15 seconds by default). Missing heartbeats indicate connectivity issues or node failures.

**Metric Cardinality:** The number of unique time series in Prometheus. High cardinality (many unique label combinations) increases storage and query costs. Monitor cardinality in production deployments.

**PromQL (Prometheus Query Language):** Query language for retrieving and aggregating metrics from Prometheus. Supports functions like `rate()`, `increase()`, `histogram_quantile()` for analysis.

**Alert Rules:** Prometheus expressions that evaluate to true/false. When an alert rule evaluates to true for the specified `for` duration, the alert fires.

**Observability Pillars:** Metrics (what's happening), logs (why it's happening), traces (how it's happening). This tutorial covered metrics and logs; tracing is covered in advanced guides.

## Next Steps

Now that you have comprehensive monitoring, here's what to explore next:

**Advanced Monitoring:**

**Alerting and Incident Response:**

**Performance Optimization:**

**Production Operations:**

**Architecture Understanding:**

## Node Disconnection and Recovery

Edge nodes can lose connectivity due to network issues, maintenance, or resource problems. Expanso handles this automatically.

### Connection States

Expanso tracks four connection states:

- **Connecting**: Node completed handshake and is establishing a stable connection
- **Connected**: Node is sending heartbeats and can receive new work
- **Disconnected**: Node missed heartbeats for 90 seconds (default) but may recover
- **Lost**: Node has been offline for 7 days (default) and is considered gone

New work only goes to Connected nodes. Nodes in the Connecting state will automatically transition to Connected once they've established a stable connection.

These thresholds are configurable defaults, set in the orchestrator's node-manager settings: `heartbeat_interval` (15s), `connected_after` (30s), `disconnect_timeout` (90s), and `lost_timeout` (7 days). If you tune them, `lost_timeout` must be greater than `disconnect_timeout`, which must be greater than `heartbeat_interval`.

### What Happens to Jobs

**Daemon jobs (long-running pipelines):**
- Running executions stay as-is and may recover when the node reconnects
- Pending executions are marked Failed immediately
- After 7 days offline, all executions are marked Failed and rescheduled to healthy nodes

**Ops jobs (one-shot tasks):**
- Executions are marked Failed after 90 seconds
- New executions are scheduled automatically when the node reconnects

### Automatic Recovery

When a disconnected node reconnects:

1. The node resumes sending heartbeats
2. It transitions back to Connected state
3. For daemon jobs, running executions continue processing
4. For ops jobs, new executions are scheduled automatically

You don't need to redeploy jobs—the system handles recovery automatically.

### Verify Node Reconnection

After a node comes back online, verify it's receiving work:

```bash
# Check node connection state
expanso-cli node list
expanso-cli node describe <node-id>
```

Look for `ConnectionState: Connected` in the output.

Check that executions are running:

```bash
# List executions on the node
expanso-cli node executions <node-id>
```

You should see executions progressing from `Pending` → `Starting` → `Running`.

### Stuck Pending Executions

Rarely, executions get stuck in `Pending` state if a node disconnects at the wrong moment during message delivery.

**Signs of stuck executions:**
- Executions show `Pending` state for more than 2 minutes
- Jobs show "deploying" state for extended periods

**Check for stuck executions:**

```bash
expanso-cli job executions <job-name>
expanso-cli execution describe <execution-id>
```

**Workaround:**

Redeploy the job:

   ```bash
   expanso-cli job stop <job-name>
   expanso-cli job delete <job-name>
# Wait 30 seconds
   expanso-cli job deploy my-pipeline.yaml
   ```

### Troubleshoot Reconnection Issues

If a node reconnects but pipelines aren't running:

**Verify the node is connected:**

```bash
expanso-cli node describe <node-id>
# Look for ConnectionState: Connected and recent LastHeartbeat
```

If disconnected, check network connectivity and that the edge agent is running:

```bash
systemctl status expanso-edge
```

**Check for executions:**

```bash
expanso-cli job executions <job-name>
expanso-cli node executions <node-id>
```

Common issues:
- **Resource constraints**: Node doesn't have enough CPU/memory
- **Bootstrap token expired**: Re-bootstrap the node
- **Label mismatch**: Node labels don't match job selector

## Troubleshooting

### Metrics Not Appearing in Prometheus

**Symptom:** Prometheus UI shows no Expanso metrics (for example, `process_cpu_utilization_ratio` returns no data).

**Diagnosis:**

```bash
# Check OTLP collector is receiving data
docker compose logs otel-collector | grep -i "received"

# Check Prometheus is scraping successfully
curl http://localhost:9091/api/v1/targets | jq
```

**Common causes:**

1. **OTLP collector not reachable:** Verify network connectivity from orchestrator/edge nodes
2. **Telemetry disabled:** Check `do_not_track: false` in configs
3. **Wrong endpoint:** Verify `endpoint` matches OTLP collector address
4. **Firewall blocking:** Ensure port 4317 (gRPC) or 4318 (HTTP) is open

**Solution:**

```bash
# Test connectivity from orchestrator
nc -zv localhost 4317

# Check telemetry logs
sudo journalctl -u expanso-orchestrator | grep -i telemetry

# Restart with debug logging
# In config: logging.level = "debug"
sudo systemctl restart expanso-orchestrator
```

### Alerts Not Firing When Expected

**Symptom:** Alert conditions are met but alerts don't show as firing.

**Diagnosis:**

```bash
# Check alert rule syntax
curl http://localhost:9091/api/v1/rules | jq

# Manually evaluate alert expression
# In Prometheus UI, run the alert's expr query
```

**Common causes:**

1. **`for` duration not elapsed:** Alert is pending, wait for full duration
2. **Syntax error in rule:** Check YAML indentation and PromQL syntax
3. **Label mismatch:** Alert expr doesn't match any actual metrics
4. **Prometheus not reloaded:** Alert rules weren't reloaded after changes

**Solution:**

```bash
# Validate alert rule syntax
promtool check rules prometheus/alerts/*.yml

# Reload Prometheus config
curl -X POST http://localhost:9091/-/reload

# Check Prometheus logs for errors
docker compose logs prometheus | grep -i error
```

### Grafana Shows "No Data"

**Symptom:** Grafana panels show "No data" despite Prometheus having metrics.

**Diagnosis:**

1. Test the query in Prometheus UI first
2. Check the time range in Grafana (top-right)
3. Verify data source is correctly configured

**Common causes:**

1. **Query syntax error:** PromQL syntax differs between Prometheus and Grafana
2. **Time range mismatch:** Data exists but not in selected time range
3. **Data source issue:** Grafana can't reach Prometheus
4. **Legend template error:** Invalid label syntax in legend

**Solution:**

```bash
# Test Prometheus data source in Grafana
# Settings → Data Sources → Expanso Prometheus → Save & Test

# Check Grafana logs
docker compose logs grafana | grep -i error

# Verify Prometheus is reachable from Grafana container
docker compose exec grafana curl http://prometheus:9090/api/v1/query?query=up
```

### High Metric Cardinality Warning

**Symptom:** Prometheus logs warn about high cardinality or memory usage.

**Diagnosis:**

```promql
# Count unique time series
count({service_name=~"expanso-.*"})

# Find metrics with most series
topk(10, count by (__name__) ({service_name=~"expanso-.*"}))

# Identify high-cardinality labels
count by (node_id, service_name) ({service_name=~"expanso-.*"})
```

**Common causes:**

1. **Many unique node IDs:** Normal for large fleets, consider sampling
2. **High-cardinality labels:** Labels with many unique values (timestamps, IDs)
3. **Metrics explosion:** New metrics added without considering cardinality impact

**Solution:**

```yaml
# In Prometheus config, add metric relabeling to drop high-cardinality labels
scrape_configs:
  - job_name: 'expanso-metrics'
    metric_relabel_configs:
      # Drop labels with high cardinality if not needed
      - source_labels: [execution_id]
        action: labeldrop
```

### Edge Node Metrics Delayed or Inconsistent

**Symptom:** Some edge nodes show stale metrics or gaps in data.

**Diagnosis:**

```bash
# Check edge node telemetry logs
ssh user@edge-node
sudo journalctl -u expanso-edge | grep -i "telemetry\|export"

# Check network latency to collector
ping -c 10 orchestrator.example.com
```

**Common causes:**

1. **Network latency:** High latency to OTLP collector
2. **Export interval too short:** Increase `export_interval` for edge nodes
3. **Collector overload:** OTLP collector can't handle export rate
4. **Batch size too small:** Increase batch size in OTLP collector config

**Solution:**

```yaml
# In edge node config, increase export interval for unstable networks
telemetry:
  export_interval: "60s"  # Increase from 15s to 60s

# In OTLP collector, increase batch size
processors:
  batch:
    timeout: 30s
    send_batch_size: 2048  # Increase from 1024
```

### Need More Help?

If you're still experiencing issues:

1. **Enable debug logging** in all components:
   ```yaml
   logging:
     level: debug
   ```

2. **Collect diagnostic information:**
   ```bash
   # Prometheus diagnostics
   curl http://localhost:9091/api/v1/status/tsdb > prom-status.json

   # OTLP collector diagnostics
   docker compose logs otel-collector > otel-debug.log
   ```

3. **Check the documentation:**

4. **Community support:**
   - Email [support@expanso.io](mailto:support@expanso.io)
   - Ask in the [community Slack](https://exso.cloud/slack)

---

## OpenTelemetry Metrics Export

# OpenTelemetry Metrics Export

Monitor your Expanso Edge nodes by exporting metrics to your observability platform using OpenTelemetry (OTLP). Track pipeline performance, resource usage, and system health across your entire edge fleet.

:::info[Where to view metrics]
The primary way to monitor your pipelines is the **Monitoring tab** in the Expanso Cloud console, which shows built-in graphs per pipeline with no setup required. This guide covers the advanced path: exporting edge-node metrics over OpenTelemetry to your own observability stack (Prometheus, Grafana, Datadog), where you run the PromQL queries shown below.
:::

## Architecture

Expanso Edge uses **OpenTelemetry Protocol (OTLP)** to push metrics to a collector, which then exports to your monitoring backend:

```
┌─────────────────┐      OTLP        ┌──────────────────┐
│ Expanso Edge    │ ──────────────> │ OpenTelemetry    │
│ Node            │   (push, gRPC)   │ Collector        │
└─────────────────┘                  └──────────────────┘
                                              │
                        ┌─────────────────────┼─────────────────────┐
                        │                     │                     │
                        ▼                     ▼                     ▼
                  ┌──────────┐          ┌─────────┐          ┌──────────┐
                  │Prometheus│          │ Grafana │          │ Datadog  │
                  │          │          │  Cloud  │          │          │
                  └──────────┘          └─────────┘          └──────────┘
```

**Why OTLP instead of Prometheus scraping?**
- ✅ Works through firewalls/NAT (push, not pull)
- ✅ Single metrics pipeline for multiple backends
- ✅ No inbound ports required on edge nodes
- ✅ Centralized collector for filtering/routing

## Quick Start

### 1. Configure Edge Node

Enable telemetry export in your edge configuration:

```yaml title="edge-config.yaml"
name: edge-node-1
data_dir: /var/lib/expanso-edge

# Enable telemetry export
telemetry:
  # OpenTelemetry Collector endpoint
  endpoint: "otel-collector.example.com:4317"
  protocol: grpc
  export_interval: 30s

  # Include Go runtime metrics
  include_go_metrics: true
  process_metrics_interval: 15s

  # Tag all metrics with these attributes
  resource_attributes:
    service.name: "expanso-edge"
    environment: "production"
    region: "us-west-2"
```

### 2. Deploy OpenTelemetry Collector

```yaml title="otel-collector-config.yaml"
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

exporters:
  # Export to Prometheus via remote write
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]
```

### 3. Query Metrics

Once in Prometheus, query edge node metrics:

```promql
# Pipeline readiness time
histogram_quantile(0.95, rate(pipeline_readiness_duration_bucket[5m]))

# Memory usage in MB
process_memory_usage_bytes / 1024 / 1024

# Pipeline errors
rate(pipeline_orchestration_errors_total[5m])
```

## Available Metrics

### Pipeline Metrics

Metrics from data pipeline execution:

| Metric | Type | Description |
|--------|------|-------------|
| `pipeline.readiness.duration` | Histogram | Time for pipeline to become ready during startup |
| `pipeline.orchestration.errors` | Counter | Number of pipeline orchestration errors |

### Pipeline Component Attributes

Pipeline metrics get tagged with attributes that let you filter and identify metrics for specific components in your data pipelines. These attributes are automatically attached to pipeline-level metrics so you can monitor and troubleshoot individual components.

#### Available Component Attributes

| Attribute | Description | Example Value | When Available |
|-----------|-------------|---------------|----------------|
| `component_id` | UUID from visual builder | `uuid-proc-123` | When component created in visual builder |
| `component_label` | User-friendly component name | `Data Filter`, `Kafka Output` | When label set in pipeline config |
| `component_name` | pipeline component type | `bloblang`, `kafka`, `http_client` | Always available |
| `component_type` | Component category | `input`, `processor`, `output` | Always available |

These attributes let you:

- Filter metrics by component type (query only processor metrics or only output metrics)
- Track specific components (monitor the component you labeled "Data Filter" across all pipelines)
- Correlate with visual builder (match metrics to components using the UUID from your visual pipeline)
- Group and aggregate (group error rates by component type or specific component names)

#### Example Queries

**Monitor the error rate for a specific component by label:**

```promql
rate(pipeline_orchestration_errors_total{component_label="Data Filter"}[5m])
```

**Group errors by component type:**

```promql
sum by (component_type) (
  rate(pipeline_orchestration_errors_total[5m])
)
```

:::tip[Component Labels for Better Observability]
Adding descriptive labels to your pipeline components makes your metrics way easier to filter and understand. Instead of tracking "root.pipeline.processors.2", you can monitor "Data Enrichment" or "PII Filter". Labels are set in your pipeline YAML with the `label` field.
:::

#### Setting Component Labels

Add labels to components in your pipeline configuration:

```yaml
pipeline:
  processors:
    - label: "Data Filter"  # This becomes component_label
      mapping: |
        root = this.filter(v -> v.status == "active")

    - label: "PII Redaction"  # This becomes component_label
      mapping: |
        root.email = this.email.re_replace_all("[^@ ]+@", "***@")
        root.ssn = this.ssn.re_replace_all("[0-9]{3}-[0-9]{2}", "***-**")
```

When these components report metrics, they'll include `component_label="Data Filter"` and `component_label="PII Redaction"`, making them easy to identify in your monitoring dashboard.

### Process Metrics

System resource metrics (collected automatically):

| Metric | Type | Description |
|--------|------|-------------|
| `process.cpu.utilization` | Gauge | CPU utilization as a ratio (0 to 1) |
| `process.memory.usage` | Gauge | Resident memory size (RSS) |
| `process.memory.virtual` | Gauge | Virtual memory size |
| `process.open_file_descriptors` | Gauge | Number of open file descriptors (Unix/Linux) |

Platform support: Linux, macOS, Windows

### Go Runtime Metrics

When `include_go_metrics: true`:

| Metric | Type | Description |
|--------|------|-------------|
| `go_goroutines_count_ratio` | Gauge | Number of active goroutines |
| `go_threads_count` | Gauge | Number of OS threads created |
| `go_max_procs` | Gauge | Current GOMAXPROCS value |
| `go_memory_heap_alloc_bytes` | Gauge | Bytes currently allocated on the heap |
| `go_gc_pause_last_seconds` | Gauge | Duration of the most recent garbage-collection pause |

### HTTP API Metrics

Metrics from the HTTP API servers on your workspace and edge nodes. Use these to monitor API usage, authentication issues, and validation errors.

#### Request Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `http.server.request.duration` | Histogram | Duration of HTTP requests in seconds |
| `http.server.request.count` | Counter | Total number of HTTP requests |
| `http.server.active_requests` | UpDownCounter | Currently active requests |
| `http.server.response.body.size` | Histogram | Size of HTTP responses in bytes |

**Labels:**
- `http.request.method` - HTTP method (GET, POST, etc.)
- `http.route` - Matched route pattern (e.g., `/api/v1/jobs/:id`)
- `http.response.status_code` - Response status code
- `error.type` - Error type for 4xx/5xx responses (e.g., `not_found`, `internal_server_error`)

#### Auth and Validation Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `http.server.auth.failures` | Counter | Authentication failures |
| `http.server.validation.failures` | Counter | Request validation failures |

**Authentication failure labels:**
- `http.request.method` - HTTP method
- `http.route` - Matched route pattern
- `auth.failure.reason` - Either `missing_token` or `invalid_token`

**Validation failure labels:**
- `http.request.method` - HTTP method
- `http.route` - Matched route pattern
- `validation.type` - Either `struct`, `submission`, or `custom`

#### Example Queries

```promql
# Average request latency by endpoint (p95)
histogram_quantile(0.95, rate(http_server_request_duration_seconds_bucket[5m]))

# Request rate by HTTP method
rate(http_server_request_count_total[5m])

# Authentication failure rate
rate(http_server_auth_failures_total[5m])

# Error rate (4xx/5xx responses)
sum(rate(http_server_request_count_total{http_response_status_code=~"4..|5.."}[5m]))
```

## Metric Filtering

Edge nodes automatically drop certain high-cardinality metrics to reduce telemetry costs and noise. You can control which metrics get filtered using `drop_metric_prefixes`.

### Default Behavior

**Edge nodes** drop these metrics by default:

| Prefix | Metrics Dropped | Reason |
|--------|-----------------|--------|
| `db.` | Database client metrics (`db.client.operation.duration`, etc.) | High cardinality from operation/table labels |
| `ncl.` | NCL messaging metrics | Internal transport, not needed for edge monitoring |
| `ncltransport.` | NCL transport metrics | Internal transport, not needed for edge monitoring |

**Orchestrators** export all metrics by default (no filtering).

### Re-enabling Metrics

To export all metrics from an edge node (useful for debugging):

```yaml title="edge-config.yaml"
telemetry:
  endpoint: "collector.example.com:4317"
  drop_metric_prefixes: []  # Empty array = keep all metrics
```

### Custom Metric Filtering

Drop additional metrics to further reduce costs:

```yaml title="edge-config.yaml"
telemetry:
  endpoint: "collector.example.com:4317"
  drop_metric_prefixes:
    - "db."
    - "ncl."
    - "ncltransport."
    - "store_gc."      # Drop GC cleanup metrics
    - "go_"            # Drop Go runtime metrics
```

### Available Metric Prefixes

| Prefix | Description | Component |
|--------|-------------|-----------|
| `process.` | Process metrics (CPU, memory, file descriptors) | All |
| `go_` | Go runtime metrics (GC, goroutines) | All (opt-in) |
| `db.` | Database client metrics | All |
| `store_gc.` | Store garbage collection | All |
| `pipeline.` | Pipeline orchestration | Edge |
| `ncl.` | NCL messaging | All |
| `ncltransport.` | NCL transport | All |
| `http.server.` | HTTP server metrics | Orchestrator |
| `evaluation.` | Evaluation metrics | Orchestrator |
| `scheduler.` | Scheduler metrics | Orchestrator |

:::tip[Cost Optimization]
If you're sending metrics to a hosted service like Grafana Cloud or Datadog, filtering unused metrics at the source can significantly reduce your telemetry costs. Start with the defaults and only re-enable metrics you actually need.
:::

## Configuration Reference

### Complete Telemetry Config

```yaml title="edge-config.yaml"
telemetry:
  # Disable all telemetry (default: false)
  do_not_track: false

  # Collector endpoint (required)
  endpoint: "collector.example.com:4317"

  # Optional path under endpoint (e.g., "/v1/metrics")
  endpoint_path: ""

  # Protocol: "grpc" (recommended, port 4317) or "http" (port 4318)
  protocol: grpc

  # Skip TLS verification (NOT recommended for production)
  insecure: false

  # How often to export metrics (default: 60s)
  export_interval: 60s

  # Custom headers for authentication
  headers:
    Authorization: "Bearer your-api-token"
    X-Custom-Header: "value"

  # Resource attributes (tags/labels applied to all metrics)
  resource_attributes:
    service.name: "expanso-edge"
    service.version: "1.0.0"
    deployment.environment: "production"
    cloud.region: "us-west-2"
    cloud.availability_zone: "us-west-2a"
    host.name: "${HOSTNAME}"

  # Include Go runtime metrics (default: false)
  include_go_metrics: true

  # Process metrics collection interval (default: 60s)
  process_metrics_interval: 60s

  # Metric prefixes to drop (default: [] for orchestrator, ["db.", "ncl.", "ncltransport."] for edge)
  drop_metric_prefixes:
    - "db."
    - "ncl."
    - "ncltransport."

  # Alternative authentication config
  authentication:
    type: "Bearer"  # or "Basic"
    token: "your-bearer-token"
    namespace: "production"
```

### Authentication

**Method 1: Headers (recommended)**

```yaml
telemetry:
  endpoint: "collector.example.com:4317"
  protocol: grpc
  headers:
    Authorization: "Bearer ${OTEL_TOKEN}"
```

**Method 2: Authentication Config**

```yaml
telemetry:
  endpoint: "collector.example.com:4317"
  protocol: grpc
  authentication:
    type: "Bearer"
    token: "${OTEL_TOKEN}"
    namespace: "production"
```

### Resource Attributes

**Resource attributes** are key-value pairs that identify the source of telemetry data. They're attached to every metric, trace, and log exported from your edge nodes, making it easy to filter and group data in your monitoring backend.

```yaml
telemetry:
  resource_attributes:
    service.name: "expanso-edge"           # Identifies the service
    deployment.environment: "production"   # Environment (dev/staging/prod)
    cloud.region: "us-west-2"              # Geographic location
    host.name: "${HOSTNAME}"               # Uses environment variable
```

**Common attributes:**

| Attribute | Description | Example |
|-----------|-------------|---------|
| `service.name` | Identifies the service | `expanso-edge` |
| `service.version` | Application version | `1.2.0` |
| `deployment.environment` | Deployment environment | `production`, `staging` |
| `cloud.region` | Cloud region | `us-west-2`, `eu-central-1` |
| `cloud.availability_zone` | Availability zone | `us-west-2a` |
| `host.name` | Hostname | `edge-node-01` |

**Tips:**
- Use consistent naming across your fleet for effective filtering
- Avoid high-cardinality values (like UUIDs) in attributes - they can bloat your metrics database
- Environment variables (`${VAR}`) are expanded at runtime

For the full list of semantic conventions, see the [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/).

---

## Monitoring Backend Setup

### Prometheus + Grafana

**OpenTelemetry Collector Config:**

```yaml title="otel-collector-config.yaml"
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
    external_labels:
      cluster: "edge-fleet-1"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]
```

**Prometheus Config:**

```yaml title="prometheus.yml"
global:
  scrape_interval: 15s

# Enable remote write receiver
# Start with: --web.enable-remote-write-receiver
```

**Grafana Dashboard Queries:**

```promql
# Pipeline readiness (p95)
histogram_quantile(0.95,
  rate(pipeline_readiness_duration_bucket{service_name="expanso-edge"}[5m])
)

# Memory usage per node
process_memory_usage_bytes{service_name="expanso-edge"} / 1024 / 1024

# Pipeline error rate
rate(pipeline_orchestration_errors_total[5m])

# CPU usage percentage
process_cpu_utilization_ratio{service_name="expanso-edge"} * 100

# Goroutines (if Go metrics enabled)
go_goroutines_count_ratio{service_name="expanso-edge"}
```

### Grafana Cloud

```yaml title="otel-collector-config.yaml"
exporters:
  prometheusremotewrite:
    endpoint: "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
    headers:
      Authorization: "Bearer ${GRAFANA_CLOUD_API_KEY}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]
```

### Datadog

```yaml title="otel-collector-config.yaml"
exporters:
  datadog:
    api:
      key: "${DD_API_KEY}"
      site: datadoghq.com

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [datadog]
```

### Elastic (ELK Stack)

```yaml title="otel-collector-config.yaml"
exporters:
  otlp/elastic:
    endpoint: "https://elastic-apm-server:8200"
    headers:
      Authorization: "Bearer ${ELASTIC_APM_TOKEN}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp/elastic]
```

### New Relic

```yaml title="otel-collector-config.yaml"
exporters:
  otlp/newrelic:
    endpoint: "https://otlp.nr-data.net:4317"
    headers:
      api-key: "${NEW_RELIC_LICENSE_KEY}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp/newrelic]
```

### Honeycomb

```yaml title="otel-collector-config.yaml"
exporters:
  otlp/honeycomb:
    endpoint: "api.honeycomb.io:443"
    headers:
      x-honeycomb-team: "${HONEYCOMB_API_KEY}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp/honeycomb]
```

## Docker Compose Example

Complete monitoring stack with edge node, collector, Prometheus, and Grafana:

```yaml title="docker-compose.yml"
version: '3.8'

services:
  expanso-edge:
    image: ghcr.io/expanso-io/expanso-edge:latest
    environment:
      - EXPANSO_EDGE_NAME=edge-docker-1
      - HOSTNAME=edge-docker-1
    volumes:
      - ./edge-config.yaml:/etc/expanso/config.yaml
      - edge-data:/var/lib/expanso-edge
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml

  prometheus:
    image: prom/prometheus:latest
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-remote-write-receiver'
    ports:
      - "9090:9090"
    volumes:
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  edge-data:
  prometheus-data:
  grafana-data:
```

## Troubleshooting

### No Metrics Appearing

1. **Enable debug logging on edge node:**

```yaml title="edge-config.yaml"
log:
  level: debug
  format: json
```

Check logs for telemetry export attempts:

```bash
journalctl -u expanso-edge | grep -i telemetry
# or
docker logs expanso-edge | grep -i telemetry
```

2. **Verify collector is receiving:**

```bash
docker logs otel-collector
# Look for: "OTLP receiver started"
```

3. **Test connectivity:**

```bash
# From edge node, verify collector is reachable
telnet collector.example.com 4317
```

### Authentication Errors

Look for authentication failures in logs:

```
"error": "failed to export metrics: rpc error: code = Unauthenticated"
```

Verify:
- Token/API key is correct
- Headers are properly formatted
- Authentication type matches collector config

### Metrics Have Wrong Names

OTLP may transform metric names. Check your monitoring backend's OTLP documentation for name transformations.

### High Memory Usage

If Go metrics are enabled and memory usage is high:

```yaml
telemetry:
  include_go_metrics: false  # Disable Go runtime metrics
  process_metrics_interval: 120s  # Reduce collection frequency
```

## Security Best Practices

1. **Always use TLS in production:**
   ```yaml
   telemetry:
     endpoint: "collector.example.com:4317"
     insecure: false  # Verify TLS certificates
   ```

2. **Use authentication:**
   ```yaml
   telemetry:
     headers:
       Authorization: "Bearer ${OTEL_TOKEN}"
   ```

3. **Network isolation:**
   - Keep collector on private network
   - Use firewall rules to restrict access to port 4317/4318

4. **Rotate credentials regularly:**
   - Use environment variables for tokens
   - Implement token rotation policy

5. **Limit resource attributes:**
   - Don't include sensitive data in attributes
   - Keep cardinality reasonable

## Performance Considerations

### Export Interval

```yaml
telemetry:
  # Lower = more frequent updates, higher overhead
  # Higher = less overhead, delayed metrics
  export_interval: 60s  # Default

  # High-frequency (more overhead):
  # export_interval: 10s

  # Low-frequency (less overhead):
  # export_interval: 120s
```

### Process Metrics Interval

```yaml
telemetry:
  # How often to collect process metrics
  process_metrics_interval: 60s  # Default

  # Reduce overhead with less frequent collection:
  # process_metrics_interval: 120s
```

### Go Metrics

```yaml
telemetry:
  # Disable if not needed (reduces overhead):
  include_go_metrics: false

  # Enable for debugging memory/GC issues:
  include_go_metrics: true
```

## Next Steps

## Platform Support

# Platform Support Matrix

**Expanso Edge Version:** v2.1+
**Last Updated:** 2026-02-03

Expanso Edge is built with Go, enabling native binary compilation for multiple operating systems and architectures. All platforms listed below are **available now**.

**[Get Started →](/getting-started)**

---

## Red Hat Certified

### RHEL 9.x / 10.x
- **Architectures:** x86_64, ARM64, ppc64le, s390x
- **Status:** ✅ Red Hat Certified
- **Container Base:** UBI9

### OpenShift 4.x
- **Versions:** 4.10, 4.11, 4.12, 4.13, 4.14, 4.15
- **Variants:** OCP, ROSA, ARO, RHOIC, Dedicated, SNO, Local
- **Status:** ✅ Red Hat Certified
- **Install:** `helm install expanso-edge oci://quay.io/expanso/helm/expanso-edge`

---

## Linux

All modern Linux distributions supported:

| Architecture | Status | Use Cases |
|--------------|--------|-----------|
| **x86_64 (amd64)** | ✅ Production | Servers, cloud VMs, OpenShift |
| **ARM64 (aarch64)** | ✅ Production | Graviton, Jetson, Raspberry Pi 4/5 |
| **ARMv7** | ✅ Available | Raspberry Pi 2/3 |

**Distributions:** RHEL, Ubuntu, Debian, CentOS Stream, Rocky Linux, AlmaLinux, Fedora, SUSE, Alpine, Arch, Gentoo, and more.

---

## macOS

| Version | Status | Use Cases |
|---------|--------|-----------|
| **ARM64 (Apple Silicon)** | ✅ Available | Development, testing, CI/CD |
| **x86_64 (Intel)** | ✅ Available | Development, testing |

Container support via OrbStack, Podman Desktop, or Docker Desktop.

---

## Windows

| Version | Status | Notes |
|---------|--------|-------|
| **x86_64** | ✅ Available | Native binary or WSL2 |
| **ARM64** | ✅ Available | Native binary or WSL2 |

---

## Android

| Architecture | Status | Use Cases |
|--------------|--------|-----------|
| **ARM64** | ✅ Available | Mobile devices, Android tablets, Termux |

---

## Cloud Platforms

All major cloud providers supported:
- ✅ **AWS** - EC2, EKS, ROSA (x86_64, Graviton ARM64)
- ✅ **Google Cloud** - Compute Engine, GKE, Tau T2A ARM64
- ✅ **Microsoft Azure** - VMs, AKS, ARO (x86_64, ARM64)
- ✅ **IBM Cloud** - Virtual Servers, IKS, RHOIC, Power Systems
- ✅ **Oracle Cloud** - Compute, OKE, Ampere A1 ARM64
- ✅ **Alibaba Cloud** - ECS, ACK

---

## Kubernetes

- **Upstream Kubernetes:** 1.24 - 1.30
- **Managed Services:** EKS, GKE, AKS, IKS, OKE, and more
- **Distributions:** OpenShift, Rancher, K3s, MicroK8s, Kind, Minikube

---

## Container Runtimes

- ✅ **Podman** 4.0+ (recommended)
- ✅ **CRI-O** 1.24+ (OpenShift default)
- ✅ **containerd** 1.6+
- ✅ **Docker** (compatible)

---

## Air-Gapped Environments

✅ Fully supported for all platforms. Requires:
- Local container registry
- Pre-downloaded binaries or Helm chart
- Offline bootstrap configuration

---

## Edge & IoT Hardware

Compatible with any device supporting the target OS and architecture:
- Raspberry Pi (all models)
- NVIDIA Jetson (Nano, Xavier, Orin)
- Intel NUC, UP Board
- Dell, Advantech, Siemens, HPE edge gateways
- Custom ARM/x86 embedded systems

---

## Container Images

**Registry:** `ghcr.io/expanso-io/expanso-edge`

**Multi-arch support:**
- linux/amd64
- linux/arm64
- linux/arm/v7

---

## Minimum Requirements

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| **CPU** | 2 cores | 4+ cores |
| **RAM** | 4 GB | 8+ GB |
| **Disk** | 20 GB | 50+ GB |
| **Network** | Internet or air-gapped config | - |

---

## Need Help?

- **Custom platform builds:** [Ask Us!](mailto:support@expanso.io?subject=Platform%20Support)
- **Community support:** [Join Slack](https://exso.cloud/slack)
- **Documentation:** [docs.expanso.io](https://docs.expanso.io)
- **Professional support:** [support@expanso.io](mailto:support@expanso.io)

---

## Certification

- ✅ Red Hat Certified (container & Helm chart)
- ✅ CNCF Kubernetes conformant
- ✅ Security scanned (Quay.io Clair)
- ✅ UBI9-based containers

## Troubleshooting

# Troubleshooting

Find solutions to common issues when running Expanso Edge.

## Quick Diagnostic Commands

```bash
# Check node status
expanso-cli node list

# Check job status
expanso-cli job list

# View execution details
expanso-cli execution list --state failed
expanso-cli execution list --state degraded

# Check job logs
expanso-cli job logs <job-id>

# Check edge process status
systemctl status expanso-edge
```

## Common Issues

| Issue | Typical Symptoms |
|-------|------------------|
| [Termux/Android Argv Duplication](./termux-android-argv-duplication) | `unknown command "expanso-edge"` errors on Termux or Android |
| [HTTP Connection Errors](./http-connection-errors) | DNS, timeout, TLS, or connection refused errors in HTTP components |
| [Permission Denied After Bootstrap](./permission-denied-after-bootstrap) | Can't write to `/var/lib/expanso/edge` or similar directories |
| [Job Shows Failed But Completed](./job-shows-failed-but-completed) | Job status shows "failed" but logs show successful completion |
| [Pipeline Progress Lost After Restart](./pipeline-progress-lost-after-restart) | Pipeline restarts from the beginning after edge process restart |
| [One-Shot Jobs Restart Unexpectedly](./one-shot-jobs-restart-unexpectedly) | Batch jobs run again after edge restart when they shouldn't |
| [Restart Events Not Tracked](./restart-events-not-tracked) | Restarts don't show up in failure metrics or logs |
| [Pipeline Stuck in Failed State](./pipeline-stuck-in-failed-state) | 409 CONFLICT errors when trying to redeploy a failed pipeline |
| [Pipeline Stuck in Degraded State](./pipeline-stuck-in-degraded-state) | Pipeline can't recover from degraded state automatically |
| [Nodes Show Connected After Restart](./nodes-show-connected-after-restart) | Nodes appear connected but jobs don't start after a workspace restart |

## Next Steps

- **[Testing & Debugging](/getting-started/testing-debugging)** - Validate and debug pipeline configurations
- **[Fleet Monitoring](/operations/monitoring/fleet-monitoring)** - Set up monitoring for your edge infrastructure
- **[Error Handling](/guides/pipelines/error_handling)** - Add retry logic and dead letter queues to pipelines

## HTTP Connection Errors

# HTTP Connection Errors

HTTP components (`http_client` input, `http` processor, `http_client` output, and `websocket`) can fail to connect due to network issues. Here are the most common errors and how to fix them.

## Unable to Resolve Host

```
unable to resolve host '<hostname>'
```

DNS lookup failed for the target hostname.

1. Verify the hostname is correct (check for typos)
2. Test DNS resolution:
   ```bash
   nslookup <hostname>
   # or
   dig <hostname>
   ```
3. Check if DNS is working for other hosts
4. Try using an IP address instead of hostname to isolate the DNS issue

## Connection Refused

```
connection refused by server at <host>
```

The remote server isn't accepting connections on the specified port.

1. Verify the server is running
2. Check you're using the correct port
3. Verify firewall rules allow the connection
4. If connecting to localhost, ensure the service is bound to the correct interface (not just `127.0.0.1` if you need external access)

## Request Timed Out

```
request to <host> timed out
```

The connection or response took too long.

1. Increase the `timeout` setting in your component config:
   ```yaml
   http_client:
     url: "http://slow-server.example.com/api"
     timeout: 30s  # increase from default 5s
   ```
2. Check for network latency or server performance issues
3. Verify the server is responding at all using `curl`

## TLS Certificate Validation Failed

```
TLS certificate validation failed for <host>
```

The server's SSL/TLS certificate is invalid, expired, or not trusted.

1. If using self-signed certificates, add them to `root_cas_file`:
   ```yaml
   http_client:
     url: "https://internal.example.com/api"
     tls:
       root_cas_file: "/path/to/ca-cert.pem"
   ```

2. For testing only, you can skip verification (not recommended for production):
   ```yaml
   http_client:
     url: "https://internal.example.com/api"
     tls:
       skip_cert_verify: true
   ```

3. Check if the certificate has expired or the hostname doesn't match

## Connection Reset

```
connection to <host> was reset by the server
```

The server abruptly closed the connection.

1. Check server logs for errors
2. Verify the server isn't overloaded or crashing
3. Look for network equipment (firewalls, load balancers) that might be terminating connections

## Connection Broken During Data Transfer

```
connection to <host> was broken during data transfer
```

The server closed the connection while data was still being sent. This typically happens when the request takes too long or exceeds size limits.

1. Check if the server has request size limits you're exceeding
2. Look for server-side timeouts closing connections early
3. Check for network issues (firewalls, proxies) terminating long-running connections
4. For large payloads, use chunked transfer encoding or smaller batches

## Job Shows Failed But Completed

# Job Shows Failed But Completed

Your job shows "failed" in `expanso-cli job list`, but the logs show it completed successfully.

## Symptoms

```bash
expanso-cli job logs j-abc123
# Shows: "Pipeline processed all 10 messages successfully"

expanso-cli job list
# Output: j-abc123   my-batch-job   failed
```

## Why This Happens

Pipelines are treated as long-running daemons. When a pipeline with finite input (like `generate` with `count: N` or file inputs) finishes processing and exits, the system marks it as "failed" instead of "completed."

This affects any pipeline designed to process a fixed amount of data and exit—batch jobs using `generate` with a count, file processors, etc.

## Verify Actual Success

Check the job logs for successful completion:

```bash
expanso-cli job logs <job-id>
```

Look for messages like "Pipeline processed N messages" or "Input exhausted, shutting down gracefully" with no errors at the end. If the logs show clean completion, your job succeeded despite the status.

:::note
This will be addressed in a future release to properly distinguish between batch jobs (finite input) and daemon jobs (long-running).
:::

## Nodes Show Connected After Restart

# Nodes Show Connected After Restart

After a workspace restart (upgrades, crashes, deployments), nodes may briefly show "Connected" even though they haven't actually reconnected yet.

## Symptoms

- Nodes appear "Connected" immediately after a workspace restart
- Jobs scheduled during the first 90 seconds may have executions stuck in Pending
- After about 90 seconds, nodes that failed to reconnect transition to "Disconnected"

## Why This Happens

Your workspace saves node state to disk. When it restarts, it loads the old "Connected" state, even though those network connections no longer exist.

To handle this, your workspace uses a **90-second grace period** after startup. During this window, it doesn't mark nodes as disconnected, giving them time to re-establish heartbeats. Nodes that don't reconnect within this window are then marked as disconnected.

## How It Self-Heals

You typically don't need to do anything—the system recovers automatically:

1. **Grace period begins**: your workspace waits 90 seconds before checking heartbeat timeouts
2. **Nodes reconnect** — Edge nodes detect the restart and re-establish their connections
3. **State stabilizes** — Nodes with fresh heartbeats stay connected; others transition to disconnected

After 90 seconds, the system accurately reflects which nodes are actually connected.

## Debugging

If you suspect connection issues after a restart:

```bash
# List nodes and their connection status
expanso-cli node list

# Check if a specific node is receiving heartbeats
expanso-cli node describe <node-id>

# Check execution states for recently deployed jobs
expanso-cli execution list --job-id <job-id>
```

If executions stay in Pending for more than 2 minutes, the node may have genuinely lost connectivity (network issues, node offline, etc.).

To verify which nodes reconnected after a restart:

```bash
# After a workspace restart, verify which nodes reconnected
expanso-cli node list

# Look for nodes still in "Connecting" or "Disconnected" state
# These nodes may have network issues preventing reconnection
```

:::tip
You don't need to wait before deploying jobs after a workspace restart. The grace period handles the transition automatically, and jobs will be scheduled to nodes once they've re-established their connections.
:::

## One-Shot Jobs Restart Unexpectedly

# One-Shot Jobs Restart Unexpectedly

One-shot jobs like batch queries and data exports may restart after the edge process restarts—even when they have no retry policy and should fail permanently if interrupted.

## Why This Happens

When the edge process restarts, it tries to resume all jobs that were running. One-shot jobs that should have "run once, no retry" behavior may incorrectly restart. This can cause duplicate side effects like repeated writes, API calls, or exports.

## Workarounds

1. **Make batch operations idempotent** — Use unique IDs, upsert operations, or check-before-execute logic so repeated runs don't create duplicates
2. **Monitor for duplicate executions** — Track execution IDs and timestamps in your output system to detect when jobs run more than once
3. **Use external locking** — Implement distributed locks to prevent duplicate execution across restarts

## Permission Denied After Bootstrap

# Permission Denied After Bootstrap

Getting errors like "can't write to `/var/lib/expanso/edge`" when running the edge agent? This usually means you bootstrapped with `sudo` but are trying to run without it.

## Why This Happens

The edge agent stores credentials in system directories that require elevated permissions:

- `/var/lib/expanso/edge` - Main data directory
- `/var/lib/expanso/edge/state` - Pipeline state
- `/var/lib/expanso/edge/metrics` - Metrics storage
- `/var/lib/expanso/edge/temp` - Temporary files
- `/var/lib/expanso/edge/auth` - Authentication credentials
- `/var/lib/expanso/edge/config.d` - Configuration files

## Solutions

### Option 1: Use sudo consistently

Run both bootstrap and run with sudo:

```bash
sudo expanso-edge bootstrap --token YOUR_TOKEN
sudo expanso-edge run
```

**Best for:** Running as a system service or system-wide installation.

### Option 2: Use --data-dir for a user directory

Specify a writable location in your home directory:

```bash
# Bootstrap with custom directory
expanso-edge bootstrap --token YOUR_TOKEN --data-dir ~/.expanso/edge

# Run with same directory
expanso-edge run --data-dir ~/.expanso/edge
```

**Best for:** Running as regular user, development, or testing.

:::tip
Set the `EXPANSO_EDGE_DATA_DIR` environment variable to avoid repeating the flag:

```bash
export EXPANSO_EDGE_DATA_DIR=~/.expanso/edge
expanso-edge bootstrap --token YOUR_TOKEN
expanso-edge run
```
:::

### Option 3: Fix permissions manually

Create directories with proper ownership:

```bash
# Create directories
sudo mkdir -p /var/lib/expanso/edge/{state,metrics,temp,auth,config.d}

# Change ownership to your user
sudo chown -R $USER:$USER /var/lib/expanso/edge

# Now you can run without sudo
expanso-edge run
```

**Best for:** Using system directory but running as regular user.

## Re-bootstrapping

If you already bootstrapped with wrong permissions, you may need to re-bootstrap:

```bash
# Option 1: Bootstrap to new location
expanso-edge bootstrap --token NEW_TOKEN --data-dir ~/.expanso/edge

# Option 2: Fix permissions and keep existing credentials
sudo chown -R $USER:$USER /var/lib/expanso/edge
expanso-edge run
```

:::note
Bootstrap tokens are time-limited and can be revoked. Generate a new token from Expanso Cloud if the current one has expired or been revoked.
:::

## Automatic Credential Detection

The edge agent detects if credentials exist in a different data directory. Instead of a generic "bootstrap required" error, you'll see where the credentials were found:

```
Error: credentials not found in /home/user/.expanso/edge, but exist in /var/lib/expanso/edge
Hint: Either run with: expanso-edge run --data-dir /var/lib/expanso/edge
      or re-bootstrap with: expanso-edge bootstrap --token ... --data-dir /home/user/.expanso/edge
```

## Pipeline Progress Lost After Restart

# Pipeline Progress Lost After Restart

When the edge process restarts (crash, `kill -9`, system reboot, or upgrade), running pipelines lose progress and restart from the beginning.

## Symptoms

A pipeline that was 90% complete restarts from 0% after an edge restart, with no indication in execution status or logs that state was lost.

## How to Check

Check if the edge process recently restarted:

```bash
# Check when edge process last started
ps aux | grep expanso-edge

# Check system logs for edge restarts
journalctl -u expanso-edge | grep -E "Started|Stopped|Killed"

# Check edge uptime
systemctl status expanso-edge
```

If the edge recently restarted and you have long-running executions, those executions likely lost progress.

## Workarounds

1. **Design pipelines to be idempotent** - Make sure reprocessing data from the beginning is safe
2. **Use external checkpointing** - Store progress in external state (database, file) and resume from there
3. **Monitor edge uptime** - Set up alerts for edge process restarts
4. **Add explicit logging** - Log progress markers so you can detect restarts in pipeline logs

## Pipeline Stuck in Degraded State

# Pipeline Stuck in Degraded State

Sometimes a pipeline gets stuck in a degraded state and won't recover on its own. This usually happens when the executor has a stale process that ignores restart signals from the scheduler.

## Diagnose the Problem

First, check if your execution is actually stuck:

```bash
# List executions by state
expanso-cli execution list --state degraded

# Check how long execution has been degraded
expanso-cli execution describe <execution-id>

# If the execution has been degraded longer than your retry backoff period
# (e.g., 10+ minutes when backoff is 5 minutes), it's likely stuck
```

## Solutions

### Stop and restart the job

```bash
# Stop the job (terminates stuck executions)
expanso-cli job stop <job-id>

# Restart the job
expanso-cli job deploy your-pipeline.yaml
```

### Restart the edge agent

If stopping the job doesn't work:

```bash
# On the edge node
sudo systemctl restart expanso-edge
```

## Monitoring and Prevention

Catch stuck executions automatically:

```bash
# Check for executions stuck in Degraded for >10 minutes
expanso-cli execution list --state degraded --format json | \
  jq '.[] | select(.updated_at < (now - 600))'
```

**To prevent this in the future:**

- Set alerts for executions stuck in Degraded state
- Use health checks and automatic rollback when available
- Design pipelines with error handling and retries

## Pipeline Stuck in Failed State

# Pipeline Stuck in Failed State

When you try to redeploy or restart a failed pipeline, you get `409 CONFLICT` errors because the edge thinks the job is still running.

:::info[Fixed in Latest Version]
This issue was fixed in the latest version of Expanso Edge. Terminal executions (Complete, Failed, Stopped) are now automatically cleaned up when starting new executions. If you're experiencing this problem, upgrade to the latest version.
:::

## Symptoms

On older versions, you'll see this sequence in the logs:

```bash
# 1. Pipeline fails with runtime error
ERR failed assignment (line 1): expected string value, got null
    execution_id=e-8fa2319e-... pipeline_id=j-465ba85c-...

# 2. Job marked as failed
INF Updating job state new_state=failed reason="Daemon execution completed unexpectedly"

# 3. Workspace retries every 30 seconds
INF Processing RunExecution request execution_id=e-2aeb1b32-...

# 4. Edge rejects with 409 CONFLICT
ERR Failed to start execution component=execution_scheduler
    error.code=CONFLICT error.http_status=409
    error.message="pipeline j-465ba85c-... is already running with a different execution"

# 5. Retry loop continues indefinitely
INF Degraded execution - backoff elapsed, retrying ...
ERR Failed to start execution ... error.code=CONFLICT
```

## Why This Happens

When a pipeline fails, the edge node keeps the failed execution in its tracking map. New execution attempts get rejected because the edge thinks the job is still running—even though the runner is in a failed state.

## Workaround

Manually stop the job before redeploying:

```bash
# Stop the stuck job
expanso-cli job stop <job-name>

# Wait a few seconds for cleanup
sleep 5

# Redeploy (with fixed configuration if needed)
expanso-cli job deploy your-pipeline.yaml
```

## Restart Events Not Tracked

# Restart Events Not Tracked

When an edge process restarts, running executions skip the normal failure flow (Running → Failed → Degraded → retry) and jump directly to Running again.

## Impact

Because restarts bypass the failure flow:

- Failure metrics aren't incremented
- Error details aren't recorded
- Retry counts don't update
- Monitoring dashboards miss these events

## Workarounds

1. **Monitor edge process health** — Track edge uptime and restart events separately from job metrics
2. **Add custom execution logging** — Log progress checkpoints that survive restarts
3. **Track execution duration anomalies** — Unusually long execution times may indicate silent restarts

## Termux/Android Argv Duplication

# Termux/Android Argv Duplication

Getting `unknown command "expanso-edge"` errors when running the edge agent on Termux or Android? This is caused by the platform's exec wrapper duplicating the binary name as the first argument.

The error message includes a hint when this pattern is detected:

```
Error: unknown command "expanso-edge"

Hint: It looks like the binary name "expanso-edge" was passed as an argument.
This can happen with Termux or Android exec wrappers that duplicate argv[0].
Try running with explicit arguments: expanso-edge run
Or set args after the binary: expanso-edge -- run
```

## Why This Happens

Some Android and Termux exec wrappers inject the binary name as an extra argument when launching programs. When you run:

```bash
expanso-edge run
```

The system might actually execute it as:

```bash
expanso-edge expanso-edge run
```

This causes the CLI to interpret `expanso-edge` as an unknown command instead of the `run` subcommand.

## Solutions

### Option 1: Use explicit argument separator

Add `--` between the binary name and your arguments:

```bash
expanso-edge -- run
expanso-edge -- bootstrap --token YOUR_TOKEN
```

**Best for:** Most users on affected platforms.

### Option 2: Create a wrapper script

Create a shell script that handles the argument passing correctly:

```bash
#!/bin/sh
exec /path/to/expanso-edge -- "$@"
```

Save this as `expanso-edge-wrapper` and use it instead of calling the binary directly.

**Best for:** Permanent fix without changing your workflow.

### Option 3: Use environment-specific workarounds

Some Termux configurations allow you to disable the exec wrapper behavior. Check your Termux settings or consult the Termux documentation for environment-specific options.

## Affected Platforms

This issue can occur on Termux, Android systems with custom exec wrappers, and some emulated Android environments. Standard Linux, macOS, and Windows installations are not affected.


## Full Reference

See https://docs.expanso.io/operations for complete documentation.