# Getting Started Reference

> Auto-generated from docs.expanso.io documentation

## Getting Started

# Getting Started

Pick a path based on how you want to start.

## How do you want to start?

**No installation required.** Use Expanso Cloud to deploy your first pipeline in 5-10 minutes.

1. Sign up for a free account
2. Deploy a node
3. Build a pipeline in the visual builder
4. See data flowing immediately

**[Start with Cloud →](/getting-started/quickstart/)**

**Install on your machine. No account needed.** Build and test pipelines before deploying anywhere.

1. Install Expanso Edge
2. Run your first pipeline from the terminal
3. Transform and route real data
4. Deploy to cloud when ready

**[Try It Without an Account →](/getting-started/try-locally/)**, or read the [Local Mode overview](/getting-started/local-mode)

:::tip[Not sure which to pick?]
**Pick Cloud** if you want to see results fast with no setup.
**Pick Local** if you want to understand how pipelines work before deploying.
:::

---

## Want to understand the concepts first?

Read about [architecture, pipelines, components, and edge processing](/getting-started/concepts/) before diving into hands-on tutorials. Takes about 20 minutes.

---

## After Getting Started

| Goal | Where to go |
|------|-------------|
| Explore 200+ inputs, processors, and outputs | [Components](/components) |
| See real-world pipeline patterns | [Use Cases](/use-cases) |
| Download and run ready-made pipelines | [Examples](/examples) |
| Master data transformations | [Bloblang Guide](/guides/bloblang/) |
| Deploy a local pipeline to cloud | [Deploy to Cloud](/getting-started/deploy-to-cloud) |

---

## Need Help?

- **[Configuration Formats and Common Mistakes](/getting-started/configuration-formats/)** - Distinguish agent settings, pipeline configuration, and deployable jobs
- **[FAQ](/getting-started/faq)** - Common questions and answers
- **[Troubleshooting](/getting-started/troubleshooting)** - Solutions to common issues
- **[Testing & Debugging](/getting-started/testing-debugging)** - Debug pipelines step by step
- **Support**: [support@expanso.io](mailto:support@expanso.io)

## Core Concepts

# Core Concepts

This section covers the key concepts behind Expanso Edge and edge-native data processing.

---

## Start Here

### [What is Expanso?](/getting-started/concepts/what-is-expanso)

What Expanso does and why edge processing matters.

**You'll learn:** Product overview, key capabilities, common use cases

---

## How It Works

### [Architecture](/getting-started/concepts/architecture)

Expanso's control plane, edge agents, and CLI.

**You'll learn:** System architecture, control plane vs edge, how components connect

### [Edge Processing](/getting-started/concepts/edge-processing)

Why process data at the edge instead of the cloud.

**You'll learn:** Edge vs cloud tradeoffs, bandwidth and latency benefits, compliance advantages

---

## Building Blocks

### [Pipelines & Jobs](/getting-started/concepts/pipelines)

Data flow definitions and how they deploy to the cloud.

**You'll learn:** Pipeline structure, YAML format, pipeline config vs job format, deployment

### [Components](/getting-started/concepts/components)

Inputs, processors, and outputs that make up pipelines.

**You'll learn:** Component types, how to use them, available options

### [Workspaces & Nodes](/getting-started/concepts/workspaces-and-nodes)

How workspaces organize your infrastructure and nodes execute pipelines at the edge.

**You'll learn:** Workspace patterns, node lifecycle, deployment options, labels and selectors

---

## What's Next?

- **[Quick Start](/getting-started/quickstart/)** - Deploy your first pipeline in minutes
- **[Local Mode](/getting-started/local-mode)** - Test pipelines locally first

## Architecture

# Architecture

Understand how Expanso's control plane and edge nodes work together.

## System Overview

```mermaid
graph TB
    Cloud[Expanso Control Plane]

    Node1[Edge Node 1]
    Node2[Edge Node 2]
    Node3[Edge Node 3]

    Input1[Data Sources]
    Input2[Data Sources]
    Input3[Data Sources]

    Output1[Destinations]
    Output2[Destinations]
    Output3[Destinations]

    Cloud -->|Deploy & Monitor| Node1
    Cloud -->|Deploy & Monitor| Node2
    Cloud -->|Deploy & Monitor| Node3

    Input1 --> Node1
    Input2 --> Node2
    Input3 --> Node3

    Node1 --> Output1
    Node2 --> Output2
    Node3 --> Output3

    style Cloud fill:#4A90E2
    style Node1 fill:#7ED321
    style Node2 fill:#7ED321
    style Node3 fill:#7ED321
```

---

## Expanso Control Plane

The central management interface hosted at cloud.expanso.io.

**Responsibilities:**
- Pipeline creation and configuration
- Node deployment and monitoring
- Metrics aggregation and visualization
- User authentication and authorization
- Workspace management

**You don't run this** - it's a managed service provided by Expanso. Enterprise customers can run an on-prem orchestrator.

---

## Edge Nodes

The runtime that executes pipelines on your infrastructure.

**Characteristics:**
- Lightweight single binary
- Runs on Linux, macOS, Windows
- Connects to your workspace over TLS, backed by the orchestrator control plane (Expanso Cloud or one you self-host)
- Executes one or more pipelines
- Reports metrics and health

**Deployment locations:**
- Bare metal servers
- Virtual machines
- Docker containers
- Kubernetes pods
- IoT devices and gateways

---

## Data Flow

### Where Data Lives

**Edge nodes**:
- Read from local data sources
- Process data locally
- Write to local or cloud destinations

**Expanso Control Plane**:
- Stores pipeline configurations
- Receives metrics and health data
- Provides configuration testing and troubleshooting workflows

Production pipeline records flow from sources through your edge nodes to the
configured destinations. Cloud receives the configuration and telemetry needed
to manage those pipelines, along with requested logs. Samples submitted for
transformation testing can also be sent to Cloud, as can configuration shared
with the pipeline assistant. Use synthetic or redacted samples when appropriate.

---

## Communication

**Node → Control Plane:**
- TLS-encrypted connection
- Sends: metrics, health, logs
- Receives: pipeline configs, commands

**Data → Data:**
- Direct connection from source to destination
- No intermediary (unless you configure one)
- Full control over data routing

---

## What's Next?

👉 **[Core Concepts](/getting-started/concepts/)** - Overview of key concepts

👉 **[Quick Start](/getting-started/quickstart/)** - Deploy your first pipeline

👉 **[Build Your First Pipeline](/getting-started/first-pipeline)** - Step-by-step tutorial

## Components

# Components

Components are the building blocks of pipelines. Expanso includes 200+ components for connecting to any system.

As of September 10, 2026, MongoDB input, output, processor, and cache components are available in nightly `v2.1.22-nightly.134.g0858e17a`. They are not included in stable `v2.1.21`, which the default installer selects. MongoDB pipelines require an edge-node release that includes these components.

## Component Types

### Inputs

Sources where data enters the pipeline.

**Categories:**
- **Messaging**: Kafka, NATS, RabbitMQ, MQTT
- **Databases**: PostgreSQL, MySQL, MongoDB (nightly only)
- **Files**: Local files, S3, SFTP, Azure Blob
- **HTTP**: Webhooks, REST APIs
- **Streams**: TCP, UDP, WebSocket
- **Special**: Generate (testing), Stdin

**Example:**
```yaml
input:
  kafka:
    addresses: ["localhost:9092"]
    topics: ["events"]
    consumer_group: "my-app"
```

[Browse all inputs →](/components/inputs)

---

### Processors

Transform, filter, and enrich data as it flows through.

**Categories:**
- **Transformation**: Mapping (Bloblang), JQ, JavaScript
- **Parsing**: JSON, CSV, XML, Avro, Protobuf, Grok
- **Filtering**: Conditional routing, deduplication
- **Enrichment**: HTTP requests, cache lookups, SQL queries
- **Aggregation**: Batching, windowing, grouping

**Example:**
```yaml
pipeline:
  processors:
    - mapping: |
        root.user = this.userId
        root.timestamp = now()
        root.message = this.msg.uppercase()
```

[Browse all processors →](/components/processors)

---

### Outputs

Destinations where processed data is sent.

**Categories:**
- **Messaging**: Kafka, NATS, RabbitMQ, MQTT
- **Databases**: PostgreSQL, MySQL, MongoDB (nightly only), Elasticsearch
- **Storage**: S3, GCS, Azure Blob, local files
- **HTTP**: REST APIs, webhooks
- **Observability**: Datadog, Splunk, Prometheus, and similar via the HTTP output
- **Special**: Stdout (debugging), Drop (discard)

**Example:**
```yaml
output:
  aws_s3:
    bucket: processed-data
    path: 'year=${!now().ts_format("2006")}/data.json'
    region: us-east-1
```

[Browse all outputs →](/components/outputs)

---

## Using Components

### Basic Usage

1. **Choose a component** from the catalog
2. **Configure it** using YAML
3. **Add to pipeline** in the appropriate section

### Configuration

Each component has:
- **Required fields**: Must be specified
- **Optional fields**: Have sensible defaults
- **Advanced options**: For fine-tuning

### Finding Components

**By category:**
- Visit [/components/inputs](/components/inputs)
- Visit [/components/processors](/components/processors)
- Visit [/components/outputs](/components/outputs)

**By search:**
- Use the component catalog search
- Filter by type, status, difficulty

---

## Component Features

### Batching

Many components support batching for efficiency:

```yaml
output:
  aws_s3:
    bucket: logs
    batching:
      count: 100
      period: 10s
```

### Error Handling

Handle failures gracefully:

```yaml
output:
  retry:
    max_retries: 3
    backoff:
      initial_interval: 1s
    output:
      kafka:
        addresses: ["localhost:9092"]
        topic: events
```

### Metrics

All components export metrics:
- Throughput (messages/sec, bytes/sec)
- Latency
- Errors
- Success/failure counts

---

## What's Next?

👉 **[Browse Components](/components)** - Explore all 200+ components

👉 **[Edge Processing](/getting-started/concepts/)** - Learn about edge benefits

👉 **[Examples](/examples)** - See real pipeline examples

## Edge Processing

# Edge Processing

Process data **where it's generated** instead of sending everything to the cloud.

## Traditional vs Edge-First

### Traditional Approach

```
Edge Device → All Data to Cloud → Process → Store
             ↑
         High cost, High latency, Privacy concerns
```

**Problems:**
- Send ALL data to cloud (expensive)
- Round-trip latency for results
- Network dependency (offline = broken)
- Sensitive data leaves premises

### Edge-First Approach

```
Edge Device → Process Locally → Filtered/Aggregated to Cloud
                    ↑
            Low cost, Low latency, Privacy preserved
```

**Benefits:**
- Filter and aggregate at source (cheaper)
- Immediate local results (faster)
- Continue processing offline (resilient)
- Keep sensitive data on-premises (compliant)

---

## Key Benefits

### 1. Reduced Bandwidth Costs

**Example**: IoT sensors generate 1GB/day of raw data:
- **Traditional**: Send 1GB/day = 30GB/month
- **Edge**: Filter 95%, send 50MB/day = 1.5GB/month

**Savings**: 95% reduction in bandwidth costs

---

### 2. Lower Latency

**Example**: Edge analytics for retail:
- **Traditional**: Edge → Cloud → Process → Results = 200ms+ round-trip
- **Edge**: Edge → Process → Results = under 10ms local

**Improvement**: 20x faster response time

---

### 3. Offline Resilience

**Example**: Factory floor processing:
- **Traditional**: Network down = No processing
- **Edge**: Network down = Continue processing, sync when reconnected

**Result**: 100% uptime for critical processing

---

### 4. Privacy & Compliance

**Example**: Healthcare data:
- **Traditional**: PHI leaves premises (compliance risk)
- **Edge**: Redact PHI at source, only send anonymized data

**Result**: Maintain compliance, reduce risk

---

## Common Patterns

### Pattern 1: Edge Filtering

Filter out noise, send only valuable data:

```yaml
# Keep only ERROR-level logs
pipeline:
  processors:
    - mapping: |
        root = if this.level != "ERROR" { deleted() }
```

**Use case**: Log aggregation (reduce volume 90%+)

---

### Pattern 2: Regional Aggregation

Aggregate locally, send summaries:

```yaml
# Aggregate locally and send only the summary
pipeline:
  processors:
    - mapping: |
        root.avg = this.values.sum() / this.values.length()
```

**Use case**: Multi-location analytics

---

### Pattern 3: Local Enrichment

Enrich with local data before sending:

```yaml
pipeline:
  processors:
    - http:
        url: "http://local-service/enrich"
        verb: POST
```

**Use case**: Add local context (device info, location)

---

### Pattern 4: PII Redaction

Remove sensitive data at the edge:

```yaml
pipeline:
  processors:
    - mapping: |
        root = this
        root.email = this.email.hash("sha256")
        root.ssn = deleted()
```

**Use case**: Compliance (GDPR, HIPAA)

---

## When to Use Edge Processing

**Good fit:**
- High data volume from edge sources
- Latency-sensitive applications
- Privacy/compliance requirements
- Unreliable network connectivity
- High bandwidth costs

**Not ideal:**
- Low data volume
- Cloud-native sources (already in cloud)
- Centralized processing required
- Edge resources constrained

---

## What's Next?

👉 **[Quickstart](/getting-started/quickstart/)** - Deploy edge pipelines in 10 minutes

👉 **[Use Cases](/use-cases)** - See edge processing in action

👉 **[Examples](/examples)** - Real pipeline configurations

## Pipelines & Jobs

# Pipelines & Jobs

A **pipeline** defines how data flows from inputs, through processors, to outputs. A **job** is how you deploy a pipeline to Expanso Cloud.

## Pipeline Structure

Every pipeline has three main sections:

```yaml
input:
  # Where data comes from
  kafka:
    addresses: ["localhost:9092"]
    topics: ["logs"]

pipeline:
  processors:
    # What to do with the data
    - mapping: |
        root.message = this.msg.uppercase()
        root.timestamp = now()

output:
  # Where data goes
  aws_s3:
    bucket: processed-logs
    path: "logs/${!timestamp_unix()}.json"
```

---

## Inputs

Sources of data:
- **Message queues**: Kafka, RabbitMQ, NATS
- **Databases**: PostgreSQL, MySQL, MongoDB (nightly only)
- **Files**: Local, S3, SFTP
- **HTTP**: Webhooks, APIs
- **Streams**: TCP, UDP, WebSocket
- **Generate**: Test data

As of September 10, 2026, MongoDB input, output, processor, and cache components are available in nightly `v2.1.22-nightly.134.g0858e17a`. They are not included in stable `v2.1.21`, which the default installer selects. MongoDB pipelines require an edge-node release that includes these components.

[Browse all inputs →](/components/inputs)

---

## Processors

Transform, filter, and enrich data:
- **Mapping**: Transform with Bloblang
- **Filter**: Drop unwanted messages
- **Parse**: JSON, CSV, XML, Avro, Protobuf
- **Enrich**: Lookup external data
- **Aggregate**: Batch, window, group

[Browse all processors →](/components/processors)

---

## Outputs

Destinations for processed data:
- **Message queues**: Kafka, RabbitMQ, NATS
- **Databases**: PostgreSQL, Elasticsearch
- **Object storage**: S3, GCS, Azure Blob
- **HTTP**: Webhooks, APIs
- **Files**: Local, S3, SFTP
- **Observability**: Datadog, Splunk, Prometheus, and similar via the HTTP output

[Browse all outputs →](/components/outputs)

---

## Simple Example

Read files, filter, output to terminal:

```yaml
input:
  file:
    paths: ["/var/log/*.log"]

pipeline:
  processors:
    - mapping: |
        # Only keep ERROR logs
        root = if !this.contains("ERROR") { deleted() }

output:
  stdout:
    codec: lines
```

---

## Multi-Output Example

Route data to different destinations:

```yaml
input:
  http_server:
    address: "0.0.0.0:8080"

pipeline:
  processors:
    - mapping: |
        root = this.parse_json()

output:
  broker:
    pattern: fan_out
    outputs:
      # Errors to Slack
      - switch:
          cases:
            - check: this.level == "ERROR"
              output:
                http_client:
                  url: "https://slack.com/webhook"

      # Metrics to a Kafka topic
      - switch:
          cases:
            - check: this.type == "metric"
              output:
                kafka:
                  addresses: ["localhost:9092"]
                  topic: metrics

      # Everything to S3
      - aws_s3:
          bucket: all-events
```

---

## Pipeline Config vs Job Format

There are two YAML formats and it's important to know when to use each:

### Pipeline config format (the processing logic)

The pipeline itself, with input/pipeline/output at the root level. This is what
`expanso-edge validate` checks, and what gets nested inside a job. It is not
something you run directly:

```yaml
input:
  file:
    paths: ["./data.json"]
pipeline:
  processors:
    - mapping: "root = this"
output:
  stdout: {}
```

### Job format (what you deploy)

Used with `expanso-cli job deploy`, for Expanso Cloud and for a local agent
alike. Wraps the pipeline config under a `config:` key with `name` and `type` at
the top level:

```yaml
name: my-file-processor
type: pipeline
config:
  input:
    file:
      paths: ["./data.json"]
  pipeline:
    processors:
      - mapping: "root = this"
  output:
    stdout: {}
```

:::tip[Converting Between Formats]
To convert a pipeline config to job format: add `name` and `type: pipeline` at the top level, then indent the entire pipeline config under `config:`. See [Deploy to Cloud](/getting-started/deploy-to-cloud) for the full walkthrough.
:::

---

## Deployment

Pipelines deploy to nodes based on:
- **Direct selection**: Choose specific nodes
- **Label selectors**: Target nodes with matching labels
- **Workspace**: All nodes in a workspace

Example with labels:

```yaml
# Deploy to production log processors only
selector:
  env: production
  role: log-processor
```

---

## What's Next?

👉 **[Components](/components)** - Learn about building blocks

👉 **[Build a Pipeline](/getting-started/first-pipeline)** - Get hands-on

👉 **[Bloblang Guide](/guides/bloblang)** - Master data transformations

## What is Expanso?

# What is Expanso?

Expanso Edge is a managed platform for deploying and orchestrating data pipelines at the edge.

## The Problem

Traditional data processing sends all data to the cloud:
- **High bandwidth costs** - Transmitting raw data
- **High latency** - Round-trip to cloud and back
- **Privacy concerns** - Sensitive data leaves premises
- **Network dependency** - Stops working offline

## The Expanso Solution

Process data **where it's generated**, at the edge:
- **Reduce bandwidth** - Filter and aggregate before sending
- **Lower latency** - Process locally, results instantly
- **Keep data local** - Sensitive data stays on-premises
- **Work offline** - Continue processing during outages

---

## Key Capabilities

### Edge-Native Processing

Data pipelines run on edge nodes, close to data sources. Only filtered/aggregated results go to the cloud.

### Managed SaaS

- No infrastructure to maintain
- Centralized control plane in the cloud
- Deploy pipelines from web UI
- Monitor fleet health and metrics

### Visual Pipeline Builder

Build data pipelines without code:
- Point and click component selection
- Live preview
- Save and deploy instantly

### 200+ Built-in Components

Connect to anything:
- Message queues (Kafka, NATS, RabbitMQ)
- Databases (PostgreSQL, MySQL; MongoDB requires the nightly release below)
- Object storage (S3, GCS, Azure Blob)
- APIs and webhooks
- Files and streams

As of September 10, 2026, MongoDB input, output, processor, and cache components are available in nightly `v2.1.22-nightly.134.g0858e17a`. They are not included in stable `v2.1.21`, which the default installer selects. MongoDB pipelines require an edge-node release that includes these components.

---

## How It Works

1. **Deploy** edge nodes to your infrastructure
2. **Build** pipelines in Expanso Cloud
3. **Deploy** pipelines to nodes
4. **Monitor** from centralized dashboard

All processing happens at the edge. Only metrics and (optionally) processed data go to the cloud.

---

## Common Use Cases

- **Log Processing**: Filter and route logs at the source
- **IoT Data**: Aggregate sensor data before cloud transmission
- **Edge Analytics**: Compute metrics locally, send summaries
- **Data Transformation**: Parse, enrich, and format at the edge
- **Compliance**: Redact PII before data leaves premises

---

## What's Next?

👉 **[Core Concepts](/getting-started/concepts/)** - Understand how the system works

👉 **[Cloud Quickstart](/getting-started/quickstart/)** - Get hands-on in 10 minutes

👉 **[Introduction](/getting-started)** - Back to Getting Started

## Workspaces & Nodes

# Workspaces & Nodes

Workspaces and nodes are how you organize and run pipelines across your infrastructure.

---

## Workspaces

A **workspace** is a logical grouping of nodes that work together.

### Purpose

Workspaces help you:
- **Organize nodes** by environment, region, or project
- **Isolate workloads** between teams or applications
- **Manage permissions** with role-based access
- **Deploy pipelines** to groups of nodes

### Common Patterns

**By Environment:**
```
├── dev-workspace        → 5 development nodes
├── staging-workspace    → 3 staging nodes
└── production-workspace → 20 production nodes
```

**By Geography:**
```
├── us-east-workspace    → Nodes in US East region
├── eu-west-workspace    → Nodes in EU West region
└── ap-south-workspace   → Nodes in Asia Pacific
```

**By Team/Project:**
```
├── team-data-workspace   → Data engineering nodes
├── team-ml-workspace     → ML inference nodes
└── iot-project-workspace → IoT sensor nodes
```

### Workspace Isolation

**What's isolated:**
- Nodes can only see other nodes in same workspace
- Pipelines deploy only to nodes in same workspace
- Metrics and logs scoped to workspace

**What's shared:**
- User accounts span workspaces
- Billing is account-level

### Creating Workspaces

Every organization starts with a **Default** workspace, created automatically at signup. Organizations can have up to 3 workspaces in total (the Default counts toward this limit); contact support if you need more.

**Via Cloud UI:**
1. Open the workspace dropdown in the top navigation
2. Click "Create Workspace"
3. Enter a name (letters, numbers, and dashes)
4. Click "Create Workspace"

---

## Nodes

An edge **node** is the lightweight runtime that executes data pipelines on your infrastructure.

### What is a Node?

A node is:
- A lightweight daemon (a single binary)
- Runs on your infrastructure (bare metal, VMs, containers, edge devices)
- Executes one or more pipelines
- Reports metrics to Expanso Cloud

### Resource Requirements

A node runs comfortably on modest hardware. Actual CPU, memory, and disk usage depend on the number of pipelines, data volume, buffering, and transformation complexity.

### Node Lifecycle

**1. Bootstrap** — Register with Expanso Cloud using a bootstrap token:

```bash
expanso-edge bootstrap --token YOUR_TOKEN
```

The node receives credentials and workspace assignment.

**2. Connect** — Establish secure TLS connection to control plane. The node registers with its assigned workspace, receives initial configuration, and begins health reporting.

**3. Execute** — Run assigned pipelines: download configs, process data, report metrics and status.

**4. Update** — Receive configuration updates as pipelines are deployed, updated, or removed.

### Deployment Options

**Bare metal / VMs:**
```bash
curl -sL https://get.expanso.io/edge/install.sh | bash
expanso-edge bootstrap --token TOKEN
```

**Docker:**
```bash
docker run ghcr.io/expanso-io/expanso-edge:latest run \
  --bootstrap-token 
```

**Kubernetes:**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: expanso-edge
spec:
  template:
    spec:
      containers:
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        env:
        - name: EXPANSO_EDGE_BOOTSTRAP_TOKEN
          value: YOUR_TOKEN
```

**Edge devices:** Raspberry Pi, IoT gateways, industrial PLCs.

### Health & Monitoring

Nodes send heartbeats every 15 seconds by default, reporting:
- **Connection status**: Connected/disconnected
- **Pipeline status**: Running/stopped/failed
- **Resource usage**: CPU, memory, disk
- **Error conditions**: Recent errors and warnings

The heartbeat interval is configurable per workspace through connectivity profiles (Stable network, Balanced, and Unreliable network).

---

## Labels and Selectors

Use **labels** on nodes to target specific groups within a workspace:

```yaml
# Node labels (set during bootstrap or via configuration)
labels:
  env: production
  region: us-east-1
  datacenter: dc-1
  role: log-processor
```

Deploy pipelines using selectors:

```yaml
# Deploy only to production log processors in us-east-1
selector:
  env: production
  region: us-east-1
  role: log-processor
```

Labels enable gradual rollouts, environment-specific deployments, role-based targeting, and monitoring filters.

---

## What's Next?

👉 **[Pipelines](/getting-started/concepts/pipelines)** - Learn about data flow definitions

👉 **[Deploy a Node](/getting-started/quickstart/)** - Get hands-on

## Configuration Formats and Common Mistakes

# Configuration Formats and Common Mistakes

Invocation and credential guidance checked against `expanso-edge` and `expanso-cli` CLI help on 2026-09-17.

> Read this first. These are the most frequent errors developers and AI
> assistants make when building with Expanso Edge. This will save you
> significant debugging time.

## 1. Pipeline Config vs Job Wrapper Format

Pipeline work uses two related YAML formats: processing configuration and
a deployable job wrapper. Agent settings are a separate configuration format
described below.

**Pipeline config** — the processing specification nested inside a job:
```yaml
input:
  subprocess:
    args: ["-F", "sensor.log"]
    name: tail
pipeline:
  processors:
    - mapping: "root = this"
output:
  stdout: {}
```
Top-level keys: `input`, `pipeline`, `output`, `buffer`

**Job wrapper** — for `expanso-cli job deploy job.yaml`:
```yaml
name: my-job
type: pipeline
config:
  input:
    subprocess:
      args: ["-F", "sensor.log"]
      name: tail
  pipeline:
    processors:
      - mapping: "root = this"
  output:
    stdout: {}
```
Top-level keys: `name`, `type`, `config` (pipeline config nested inside)

**Decision tree:**
- Deploying to Expanso Cloud → Job wrapper + `expanso-cli job deploy`
- Running standalone on one machine → `expanso-edge run --local`, then submit a job wrapper to its local API
- Running in local API mode → Job wrapper + `expanso-cli job deploy`

To convert: indent the pipeline config under a `config:` key, then add
`name` and `type: pipeline` at the top level.

Validate before deploying: `expanso-cli job validate my-job.yaml --offline`

## 2. Cloud CLI Endpoint URL

The `expanso-cli` endpoint for Expanso Cloud is NOT `cloud.expanso.io`.
The correct format is:

```
https://<workspace-id>.<region>.cloud.expanso.io:9010
```

To find your workspace ID:
- In cloud.expanso.io, open the workspace: the ID is the **ID** badge at the
  top of the page
- Or check an edge node's config:
  `cat /var/lib/expanso/edge/config.d/50-connection.yaml`

Example CLI setup:
```bash
expanso-cli profile save prod \
  --endpoint https://abc123xyz.us2.cloud.expanso.io:9010 \
  --api-key exp_ak_YOUR_KEY \
  --select
```

Create the API key in Expanso Cloud under your workspace **Keys** tab
(**API Keys → Create API Key**). The workspace **API Access** dialog shows your
endpoint and generates this exact command for you.

Check it worked:
```bash
expanso-cli status
```

## 3. Bootstrap Tokens and API Keys Are Different Credentials

They are not interchangeable, and `expanso-cli` has no `--auth-token` flag.

| Credential | Prefix | Used by | For |
|---|---|---|---|
| Bootstrap token | `exp_bk_` | `expanso-edge bootstrap --token` | Registering an edge node, once |
| API key | `exp_ak_` | `expanso-cli --api-key`, or `Authorization: Bearer` | Authenticating CLI and API calls |

A bootstrap token is exchanged during `expanso-edge bootstrap` for node
credentials stored in the node's data directory; it never authenticates
`expanso-cli`. Get a bootstrap token from your workspace **Keys** tab under
**Bootstrap Tokens**, and an API key from the same tab under **API Keys**.

Enrolling a node directly against an orchestrator with `expanso-edge enroll`
uses a third form, an orchestrator bootstrap token (`exp_obk_`).

## 4. Run a Pipeline End to End

Pipelines run as jobs submitted to a control plane. Prerequisites: an Expanso
Cloud workspace, an API key (`exp_ak_...`), a bootstrap token (`exp_bk_...`), and
a host to run the edge agent on.

```bash
# 1. Start an edge node, using a bootstrap token
docker run -d \
  ghcr.io/expanso-io/expanso-edge:latest run \
  --bootstrap-token exp_bk_YOUR_TOKEN

# 2. Point the CLI at your workspace with an API key, not the bootstrap token
#    (copy the endpoint from the workspace's API Access dialog)
expanso-cli profile save prod \
  --endpoint https://WORKSPACE_ID.us2.cloud.expanso.io:9010 \
  --api-key exp_ak_YOUR_KEY --select
expanso-cli status

# 3. Confirm the node registered
expanso-cli node list

# 4. Create a job file (job wrapper format)
cat > my-job.yaml << 'EOF'
name: hello-world
type: pipeline
config:
  input:
    generate:
      mapping: 'root.message = "hello from edge"'
      interval: 1s
  pipeline:
    processors:
      - log:
          message: '${! content() }'
  output:
    stdout: {}
EOF

# 5. Check the spec, then deploy it
expanso-cli job validate my-job.yaml --offline
expanso-cli job deploy my-job.yaml

# 6. Confirm it is executing, not merely stored
expanso-cli job describe hello-world
expanso-cli execution list --job-id JOB_ID
expanso-cli job logs hello-world
```

`job deploy` returns once the control plane has stored the job, not once a node
has picked it up, so a clean deploy is not evidence of execution. An empty
`execution list` means nothing is running it — check that a node is connected and
that the job's `selector` matches its labels.

Offline, with no account and no node, `expanso-edge validate pipeline.yaml`
checks a bare pipeline config's syntax and component types. It does not check
job-level fields and does not guarantee the control plane will accept the job.

See [Deploy to Cloud](/getting-started/deploy-to-cloud/) for the walkthrough.

## 5. Edge v2 --config Flag

In Edge v2, the `--config` flag loads **agent** configuration, not pipeline
configuration. If you pass a pipeline YAML to `--config`, the pipeline
keys (`input`, `pipeline`, `output`, `buffer`) are logged as "Unknown
configuration fields detected" and silently ignored.

There is no invocation that runs a bare pipeline config. Every execution path
goes through a job wrapper submitted with `expanso-cli job deploy`.

## 6. Running Without Expanso Cloud

Expanso Edge also runs standalone. `expanso-edge run --local` starts an agent
with no control-plane connection, serving its own API on port `9010`, and the
same job wrapper is submitted to it with
`expanso-cli job deploy --endpoint http://localhost:9010 job.yaml`.

Standalone mode has no workspace connection, so multi-node coordination and
fleet management do not apply. See [Local Mode](/getting-started/local-mode/) for
what it does cover.

## Deploy to Cloud

# Deploy to Cloud

This guide walks you through deploying a pipeline to Expanso Cloud. You'll
configure the CLI, write a job file, and deploy it to your connected edge
nodes.

:::info[Prerequisites]
- An Expanso Cloud account ([cloud.expanso.io](https://cloud.expanso.io))
- A bootstrap token (from your workspace **Keys** tab, under **Bootstrap Tokens**)
- An edge node installed and connected to your workspace — see [Installation](/getting-started/installation) for setup instructions
- `expanso-cli` installed (`curl -fsSL https://get.expanso.io/cli/install.sh | bash`)
:::

## Step 1: Configure the CLI

The CLI needs your cloud endpoint and an API key.

**Find your endpoint:** Your CLI endpoint follows this pattern:

```
https://<workspace-id>.<region>.cloud.expanso.io:9010
```

To find your workspace ID:
- **Option A:** Open the workspace in [cloud.expanso.io](https://cloud.expanso.io). The ID is the **ID** badge at the top of the page.
- **Option B:** Read it from a running edge node's config:

```bash
cat /var/lib/expanso/edge/config.d/50-connection.yaml
# Look for: address: nats://<workspace-id>.<region>.cloud.expanso.io:4222
# Use the same workspace-id but with port 9010 for the CLI
```

**Save a profile:**

Create an API key from your workspace **Keys** tab (**API Keys → Create API Key**). The **API Access** dialog on the workspace Overview also shows your endpoint and generates this command for you.

```bash
expanso-cli profile save my-cloud \
  --endpoint https://YOUR_WORKSPACE_ID.us2.cloud.expanso.io:9010 \
  --api-key exp_ak_YOUR_KEY \
  --select
```

:::note
The bootstrap token (`exp_bk_...`) only registers edge nodes; it cannot authenticate the CLI. Use an API key (`exp_ak_...`) instead.
:::

## Step 2: Verify the Node Connected

```bash
expanso-cli node list
```

You should see your node with state `connected`:

```
 ID       │ NAME      │ STATE     │ VERSION
 abc12345 │ my-edge   │ connected │ v2.1.7
```

If your node doesn't appear, check that it started successfully and can
reach your workspace. See [Installation](/getting-started/installation)
for troubleshooting.

## Step 3: Create a Job File

Pipelines deployed via `expanso-cli` use the **job wrapper format** — the
pipeline config is nested under a `config:` key with `name` and `type` at
the top level.

```yaml title="hello-world-job.yaml"
name: hello-world
type: pipeline
config:
  input:
    generate:
      mapping: |
        root.message = "hello from the edge"
        root.timestamp = now()
      interval: 5s
  pipeline:
    processors: []
  output:
    stdout: {}
```

:::warning[Pipeline Config vs Job Format]
Do **not** pass pipeline config format (with `input`/`pipeline`/`output` at the
root level) to `expanso-cli job deploy`. That format is the pipeline itself: it
is what `expanso-edge validate` checks and what gets nested under `config:` here.
Nothing runs it directly — in particular `expanso-edge run --config` loads
**agent** settings and will ignore it. See
[Build Your First Pipeline](/getting-started/first-pipeline)
for the pipeline config format.
:::

**Validate before deploying:**

```bash
expanso-cli job validate hello-world-job.yaml --offline
```

## Step 4: Deploy

```bash
expanso-cli job deploy hello-world-job.yaml
```

Output:
```
Job 'hello-world' created successfully in namespace ''
```

## Step 5: Verify

```bash
# Check job status
expanso-cli job describe hello-world

# List executions on nodes
expanso-cli execution list --job-id $(expanso-cli job describe hello-world 2>&1 | grep '^ID' | awk '{print $NF}')
```

## Updating a Pipeline

To update a running pipeline, edit the job file and redeploy:

```bash
expanso-cli job deploy hello-world-job.yaml --force
```

The `--force` flag applies the deploy even if the spec hasn't changed. To
restart a job with the spec it already has, use
`expanso-cli job rerun hello-world` instead.

## Converting Pipeline Config to Job Format

If you have an existing pipeline config file — input/pipeline/output at the root
level — convert it to job format by:

1. Adding `name` and `type: pipeline` at the top level
2. Indenting the entire pipeline config under a `config:` key

**Before (pipeline config):**
```yaml
input:
  file:
    paths: ["./data.json"]
pipeline:
  processors:
    - mapping: "root = this"
output:
  stdout: {}
```

**After (job wrapper):**
```yaml
name: my-file-processor
type: pipeline
config:
  input:
    file:
      paths: ["./data.json"]
  pipeline:
    processors:
      - mapping: "root = this"
  output:
    stdout: {}
```

## Next Steps

- [Build Your First Pipeline](/getting-started/first-pipeline) — step-by-step pipelines deployed as Cloud jobs
- [Local Mode](/getting-started/local-mode) — run Edge as a local API server
- [Components](/components) — explore 200+ inputs, processors, and outputs

## Frequently Asked Questions

<head>
  <script type="application/ld+json">
    {JSON.stringify({
      '@context': 'https://schema.org',
      '@type': 'FAQPage',
      mainEntity: [
        {
          '@type': 'Question',
          name: 'How long does it take to set up Expanso?',
          acceptedAnswer: {
            '@type': 'Answer',
            text: 'You can have your first pipeline running in about 5 minutes. This includes creating a cloud account, deploying a node, and building a simple pipeline. The edge agent installation and bootstrap process typically completes in under a minute.',
          },
        },
        {
          '@type': 'Question',
          name: 'Do I need to install any software on my infrastructure?',
          acceptedAnswer: {
            '@type': 'Answer',
            text: 'Yes, you need to install the Expanso Edge agent on your infrastructure. It\'s a lightweight binary that runs on Linux, macOS, or Windows. You can also run it in Docker containers. The edge agent is the only component you need to install - everything else is managed through Expanso Cloud.',
          },
        },
        {
          '@type': 'Question',
          name: 'Can I test Expanso without deploying to production?',
          acceptedAnswer: {
            '@type': 'Answer',
            text: 'Absolutely! You can use Local Mode to run pipelines standalone without any cloud connection. This is perfect for development and testing. You can also deploy nodes to non-production environments and test pipelines there before promoting to production.',
          },
        },
        {
          '@type': 'Question',
          name: 'What happens if my node goes offline?',
          acceptedAnswer: {
            '@type': 'Answer',
            text: 'If a node loses its connection to Expanso Cloud, the cloud marks it as disconnected and stops sending it new jobs, but the pipelines already running on that node keep executing autonomously at the edge. When the node reconnects, it reconverges its state with the control plane. Only if the machine itself is powered off or restarted do its pipelines stop; they resume automatically once it is back online and reconnected.',
          },
        },
        {
          '@type': 'Question',
          name: 'How do I update my pipelines after deployment?',
          acceptedAnswer: {
            '@type': 'Answer',
            text: 'You can update pipelines anytime through the Expanso Cloud UI. Simply edit the pipeline configuration and redeploy. The node will receive the updated configuration and apply it automatically, typically with zero downtime using rolling updates.',
          },
        },
      ],
    })}
  </script>
</head>

# Frequently Asked Questions

Common questions about getting started with Expanso.

---

## Setup & Installation

<details>
<summary><strong>How long does it take to set up Expanso?</strong></summary>

You can have your first pipeline running in about 5 minutes. This includes creating a cloud account, deploying an agent, and building a simple pipeline. The agent installation and bootstrap process typically completes in under a minute.

</details>

<details>
<summary><strong>Do I need to install any software on my infrastructure?</strong></summary>

Yes, you need to install the Expanso Edge agent on your infrastructure. It's a lightweight binary that runs on Linux, macOS, or Windows. You can also run it in Docker containers. The edge agent is the only component you need to install - everything else is managed through Expanso Cloud.

</details>

<details>
<summary><strong>Can I test Expanso without deploying to production?</strong></summary>

Absolutely! You can use [Local Mode](/getting-started/local-mode) to run pipelines standalone without any cloud connection. This is perfect for development and testing. You can also deploy nodes to non-production environments and test pipelines there before promoting to production.

</details>

---

## Operations

<details>
<summary><strong>What happens if my node goes offline?</strong></summary>

If a node loses its connection to Expanso Cloud, the cloud marks it as disconnected and stops sending it new jobs, but the pipelines already running on that node keep executing autonomously at the edge. When the node reconnects, it reconverges its state with the control plane. Only if the machine itself is powered off or restarted do its pipelines stop; they resume automatically once it is back online and reconnected.

</details>

<details>
<summary><strong>How do I update my pipelines after deployment?</strong></summary>

You can update pipelines anytime through the Expanso Cloud UI. Simply edit the pipeline configuration and redeploy. The node will receive the updated configuration and apply it automatically, typically with zero downtime using rolling updates.

</details>

---

## Need More Help?

- **[Troubleshooting Guide](/getting-started/troubleshooting)** - Solutions to common issues
- **[Core Concepts](/getting-started/concepts/)** - Understand how Expanso works
- **Support**: Contact us at support@expanso.io

## Build Your First Pipeline

# Build Your First Pipeline

Write a pipeline, deploy it to Expanso Cloud, and watch it run on a connected edge node. Start with a simple "Hello World" and progressively work up to processing real system data.

This series follows the **Expanso Cloud** route throughout: you author pipeline config locally, wrap it in a job spec, and deploy it with `expanso-cli job deploy`. Expanso Edge also runs standalone, but that is a separate route documented in [Local Mode](/getting-started/local-mode) — follow one or the other, not a mix of both.

## What You'll Learn

This tutorial covers **4 progressive examples**, each building on the last:

1. **[Hello World](/getting-started/first-pipeline/hello-world)** - The simplest possible pipeline
2. **[Make a Change](/getting-started/first-pipeline/make-a-change)** - Modify and see updates instantly
3. **[Data Transformation](/getting-started/first-pipeline/data-transformation)** - Convert sensor readings with calculations
4. **[Real Data](/getting-started/first-pipeline/real-data)** - Read and process live system logs

**Time required**: 10 minutes total (or 2-3 minutes per step)

---

## Prerequisites

- Linux, macOS, or Windows
- Terminal access
- An Expanso Cloud workspace, a saved `expanso-cli` profile with an API key, and
  at least one connected edge node. That one-time setup is in
  [Deploy to Cloud](/getting-started/deploy-to-cloud).

---

## Install Expanso Edge and expanso-cli

Expanso Edge is the runtime that executes your pipelines, and `expanso-cli` is
how you deploy jobs to your workspace. You need both:

```bash
# Install Expanso Edge
curl -fsSL https://get.expanso.io/edge/install.sh | bash

# Install expanso-cli
curl -fsSL https://get.expanso.io/cli/install.sh | bash

# Verify installation
expanso-edge version
expanso-cli version
```

```bash
# Install Expanso Edge
curl -fsSL https://get.expanso.io/edge/install.sh | bash

# Install expanso-cli
curl -fsSL https://get.expanso.io/cli/install.sh | bash

# Verify installation
expanso-edge version
expanso-cli version
```

```powershell
# Windows: Use Docker (PowerShell installer not yet available)
docker pull ghcr.io/expanso-io/expanso-edge:latest

# Or use WSL2 for both binaries
wsl curl -fsSL https://get.expanso.io/edge/install.sh | bash
wsl curl -fsSL https://get.expanso.io/cli/install.sh | bash
```

:::info[Windows Users]
See the [Installation Guide](/getting-started/installation) Windows tab for detailed setup options including Docker and WSL2.
:::

:::warning[The Agent Reads Files as Its Own User]
`expanso-edge` runs under the user's context, and will only have access to files
that that user explicitly has access to. It is recommended to run `expanso-edge`
under a dedicated user or group. If you see "Failed to create directories", you
may not have the necessary permissions.

Neither binary runs a pipeline from a bare config file. `expanso-edge run
--config` loads **agent** settings, as explained in
[Configuration Formats and Common Mistakes](/getting-started/configuration-formats/);
pipelines execute as jobs you deploy with `expanso-cli job deploy`.
:::

---

## Ready to Start?

👉 **[Begin with Step 1: Hello World](/getting-started/first-pipeline/hello-world)**

## Step 3: Data Transformation

# Step 3: Data Transformation

Now let's add some real processing. Imagine you have IoT sensors reporting temperature in Celsius, but you need Fahrenheit. This pipeline simulates sensor data and converts it.

---

## Create the Pipeline

Create `sensor-pipeline.yaml`:

```yaml
input:
  generate:
    interval: 1s
    mapping: |
      root.device_id = "edge-sensor-01"
      root.temp_c = random_int(min: 15, max: 35)
      root.humidity = random_int(min: 30, max: 80)

pipeline:
  processors:
    - mapping: |
        root = this
        root.temp_f = (this.temp_c * 1.8) + 32
        root.processed_at = now()

output:
  stdout:
    codec: lines
```

Check the config, then wrap it in a job and deploy it the same way as
[Step 1](/getting-started/first-pipeline/hello-world#run-it):

```bash
expanso-edge validate sensor-pipeline.yaml
```

Create `sensor-job.yaml` with `name: sensor-pipeline` and `type: pipeline` at the
top level, and the pipeline config above indented under `config:`. Then deploy it:

```bash
expanso-cli job deploy sensor-job.yaml
```

Output:

```json
{"device_id":"edge-sensor-01","humidity":65,"processed_at":"2024-12-26T10:05:00Z","temp_c":22,"temp_f":71.6}
{"device_id":"edge-sensor-01","humidity":48,"processed_at":"2024-12-26T10:05:01Z","temp_c":28,"temp_f":82.4}
{"device_id":"edge-sensor-01","humidity":72,"processed_at":"2024-12-26T10:05:02Z","temp_c":19,"temp_f":66.2}
```

---

## Understanding the Pipeline

```
[Generate Sensor Data] → [Transform: Add temp_f] → [Print to Terminal]
     (input)                  (processor)              (output)
```

**Key concepts:**
- **`pipeline.processors`** - An array of transformations applied in order
- **`root = this`** - Keep all existing fields
- **`root.temp_f = ...`** - Add a new calculated field
- **Bloblang math** - Standard operators work: `*`, `+`, `/`, `-`

---

## What's Next?

👉 **[Step 4: Process Real Data](/getting-started/first-pipeline/real-data)**

## Step 1: Hello World

# Step 1: Hello World

Let's start with the simplest possible pipeline—one that just prints "Hello, World!" every second.

---

## Create the Pipeline

Create a file called `hello.yaml`:

```yaml
input:
  generate:
    interval: 1s
    mapping: |
      root.message = "Hello, World!"

pipeline:
  processors:
    - log:
        message: '${! content() }'

output:
  stdout: {}
```

Check it before you run anything. This is offline and needs no node, no
account and no network:

```bash
expanso-edge validate hello.yaml
```

---

## Run It

`hello.yaml` is a **pipeline config**. Expanso executes pipelines as **jobs**, so
wrap it in a job spec that gives it a name and a type:

```yaml title="hello-job.yaml"
name: hello-world
type: pipeline
config:
  input:
    generate:
      interval: 1s
      mapping: |
        root.message = "Hello, World!"
  pipeline:
    processors:
      - log:
          message: '${! content() }'
  output:
    stdout: {}
```

Then deploy it to Expanso Cloud, which schedules it onto a connected edge node
and keeps it managed by the control plane. This series is written against that
route.

**Prerequisites:** an Expanso Cloud workspace, a saved CLI profile with an API
key, and at least one connected edge node. One-time setup is in
[Deploy to Cloud](/getting-started/deploy-to-cloud).

```bash
expanso-cli job deploy hello-job.yaml
expanso-cli job logs hello-world
```

:::note[Without Expanso Cloud]
Expanso Edge also runs standalone: `expanso-edge run --local` starts an agent
with no control-plane connection and its own API on port `9010`, which you can
deploy the same job spec to. That mode is documented separately in
[Local Mode](/getting-started/local-mode) — the later steps here assume the Cloud
route, so follow Local Mode's own pages rather than mixing the two.
:::

---

## Verify It Worked

`job logs` streams the pipeline's log. Every second the `log` processor adds a
line that ends with the message. Each line starts with the node, a timestamp
and labels (shortened to `{…}` here):

```text
[my-node]  2026-10-02T18:11:08.721Z {…} {"message":"Hello, World!"}
[my-node]  2026-10-02T18:11:09.721Z {…} {"message":"Hello, World!"}
```

Lines about writing to `stdout` appear in between. The `stdout` output itself
prints to the agent's terminal on the node, not to `job logs`.

If nothing appears, the job was accepted but is not executing. `job deploy`
returns after the control plane stores the job, not after a node picks it up, so
check the assignment:

```bash
expanso-cli job describe hello-world
expanso-cli node list
```

An empty node list means no node is connected to run it. More failure modes are
in [Testing & Debugging](/getting-started/testing-debugging).

Stop the job when you are done:

```bash
expanso-cli job stop hello-world
```

:::warning[`--config` does not run a pipeline]
`expanso-edge run --config` loads **agent** settings, not pipeline YAML. Passing a
pipeline config to it reports the `input`, `pipeline` and `output` keys as unknown
fields and ignores them: the agent starts and your pipeline never runs.
:::

---

## What Just Happened?

| Component | What It Does |
|-----------|--------------|
| `input.generate` | Creates data from nothing (no external source needed) |
| `interval: 1s` | Generates a new message every 1 second |
| `mapping` | Uses [Bloblang](/guides/bloblang) to define the message structure |
| `log` processor | Writes each message to the pipeline's log, which `job logs` streams |
| `output.stdout` | Prints each message to the agent's terminal on the node |

---

## What's Next?

👉 **[Step 2: Make a Change](/getting-started/first-pipeline/make-a-change)**

## Step 2: Make a Change

# Step 2: Make a Change

Now let's modify the pipeline to see how quickly you can iterate.

---

## Update the Pipeline

Update `hello.yaml`:

```yaml
input:
  generate:
    interval: 1s
    mapping: |
      root.message = "Hello, World!"
      root.timestamp = now()
      root.count = counter()

pipeline:
  processors:
    - log:
        message: '${! content() }'

output:
  stdout: {}
```

Copy the changed pipeline config under the `config:` key of `hello-job.yaml`, the
job spec you wrote in
[Step 1](/getting-started/first-pipeline/hello-world#run-it), then redeploy:

```bash
expanso-cli job deploy hello-job.yaml
```

Deploying an existing job name updates that job in place and the control plane
re-evaluates it, so you do not stop it first. `--force` applies the deploy even
when the spec is unchanged; to restart a job with the spec it already has, use
`expanso-cli job rerun hello-world`.

Now the lines from `expanso-cli job logs hello-world` end with:

```json
{"count":1,"message":"Hello, World!","timestamp":"2024-12-26T10:00:00.123Z"}
{"count":2,"message":"Hello, World!","timestamp":"2024-12-26T10:00:01.456Z"}
{"count":3,"message":"Hello, World!","timestamp":"2024-12-26T10:00:02.789Z"}
```

---

## What Changed?

- **`now()`** - A Bloblang function that returns the current timestamp
- **`counter()`** - A function that increments with each message

Try changing the message text or interval and redeploy. Each deploy creates a new
job version, so `expanso-cli job versions hello-world` shows the history and
`expanso-cli job rollback` returns to an earlier one.

---

## What's Next?

👉 **[Step 3: Data Transformation](/getting-started/first-pipeline/data-transformation)**

## Step 4: Process Real Data

# Step 4: Process Real Data

Now let's process real, live data from your system. We'll tail a log file that changes continuously.

:::info[Permission Note]
The pipeline reads the log file as whichever user runs the edge agent, not as
whoever deploys the job. If you get a permission error:
- Give that user read access to the file, or start the agent as a user that
  already has it
- Or choose a log file you have access to (like application logs)
:::

---

## Create the Pipeline

Create `logs-pipeline.yaml`:

```yaml
input:
  file:
    paths: [/var/log/syslog]
    codec: lines

pipeline:
  processors:
    - mapping: |
        root.raw = this
        root.length = this.length()
        root.processed_at = now()

output:
  stdout:
    codec: lines
```

```yaml
input:
  file:
    paths: [/var/log/system.log]
    codec: lines

pipeline:
  processors:
    - mapping: |
        root.raw = this
        root.length = this.length()
        root.processed_at = now()

output:
  stdout:
    codec: lines
```

```yaml
input:
  file:
    paths: [C:\Windows\System32\log.txt]
    codec: lines

pipeline:
  processors:
    - mapping: |
        root.raw = this
        root.length = this.length()
        root.processed_at = now()

output:
  stdout:
    codec: lines
```

Wrap it in a job and deploy it as in
[Step 1](/getting-started/first-pipeline/hello-world#run-it): create
`logs-job.yaml` with `name: logs-pipeline` and `type: pipeline` at the top level,
and the pipeline config from your platform's tab above indented under `config:`.

```bash
expanso-cli job deploy logs-job.yaml
```

`paths` is read on the node that runs the job, not on the machine you deploy
from, so the path must exist on that node — pick the tab that matches the node's
operating system, not your laptop's.

You'll see new log entries appear in real-time as your system generates them:

```json
{"length":142,"processed_at":"2024-12-26T10:10:00Z","raw":"Dec 26 10:10:00 myhost kernel: ..."}
{"length":98,"processed_at":"2024-12-26T10:10:01Z","raw":"Dec 26 10:10:01 myhost systemd: ..."}
```

---

## What's Happening?

- **`file` input** - Tails the file, reading new lines as they're written
- **`codec: lines`** - Treats each line as a separate message
- **Real-time processing** - As logs are written, they flow through your pipeline

---

## Try Adding a Filter

Want to only see error messages? Add a filter processor:

```yaml
input:
  file:
    paths: [/var/log/syslog]
    codec: lines

pipeline:
  processors:
    # Only keep lines containing "error" (case-insensitive)
    - mapping: |
        root = if !this.lowercase().contains("error") {
          deleted()
        }

    # Add metadata
    - mapping: |
        root.raw = this
        root.severity = "ERROR"
        root.processed_at = now()

output:
  stdout:
    codec: lines
```

---

## Recap: What You've Learned

| Step | Concept | Key Takeaway |
|------|---------|--------------|
| 1 | Hello World | Simplest pipeline: generate → stdout |
| 2 | Make a Change | Fast iteration, Bloblang functions |
| 3 | Transformation | Processors add/modify fields, do calculations |
| 4 | Real Data | File input tails logs in real-time |

You now understand the core pipeline pattern:

```
[Input] → [Processor(s)] → [Output]
```

---

## Next Steps

Now that you've built and deployed pipelines:

- **[Test and Debug](/getting-started/testing-debugging)** - Validate configurations and troubleshoot issues
- **[Explore Components](/components)** - Browse 200+ inputs, processors, and outputs
- **[Learn Bloblang](/guides/bloblang)** - Master the transformation language
- **[Deploy to Production](/getting-started/quickstart/)** - Set up Expanso Cloud and deploy pipelines to your infrastructure

---

## Tips

**Start Simple**: Test each part of your pipeline separately before combining them.

**Use `stdout` for Debugging**: Always output to `stdout` first to see exactly what's happening.

**Limit Test Data**: Use `count` to produce a fixed number of messages:

```yaml
input:
  generate:
    count: 5  # Generate exactly 5 messages then stop
    interval: ""  # As fast as possible
    mapping: |
      root = {"test": "data"}
```

**Fast Iteration**: Edit the YAML → `expanso-cli job deploy` again. Redeploying an
existing job name updates it in place, so there is no stop step in between.

## Installation

# Installation

Install Expanso Edge to run data pipelines on your infrastructure.

## System Requirements

Resource use depends on your pipelines, the node hardware, and the configuration. Measure CPU and memory on the machine you plan to use. The agent binary needs about 250 MB of disk space.

**Supported platforms:**
Linux, macOS, Windows, and Docker.

For a comprehensive list of supported operating systems, architectures, cloud platforms, and container runtimes, see [Platform Support](/references/platform-support).

---

## Install Expanso Edge

```bash
# Install Expanso Edge
curl -fsSL https://get.expanso.io/edge/install.sh | bash

# Verify installation
expanso-edge version
```

**Expected output:**
```
v2.1.x
```

```bash
# Install prerequisites
pkg install -y curl ca-certificates

# Install Expanso Edge into the Termux prefix
export EXPANSO_INSTALL_DIR="$PREFIX/bin"
curl -fsSL https://get.expanso.io/edge/install.sh | bash

# Verify installation
expanso-edge version
```

:::tip[Termux Troubleshooting]
Ensure `ca-certificates` is installed so TLS downloads succeed.

If bootstrap fails on mobile networks, try:
`EXPANSO_EDGE_BOOTSTRAP_FORCE_IPV4=1 expanso-edge bootstrap --token YOUR_TOKEN`
`EXPANSO_EDGE_BOOTSTRAP_NO_PROXY=1 expanso-edge bootstrap --token YOUR_TOKEN`
:::

:::warning[Windows Installation]
The automated PowerShell installer is not yet available. For Windows installation:

**Option 1: Use Docker** (Recommended)
```powershell
# Pull and run Expanso Edge with your bootstrap token
docker pull ghcr.io/expanso-io/expanso-edge:latest
docker run -d `
  --name expanso-edge `
  --restart unless-stopped `
  -e EXPANSO_EDGE_BOOTSTRAP_TOKEN=YOUR_BOOTSTRAP_TOKEN `
  ghcr.io/expanso-io/expanso-edge:latest
```

This is the same command as the **Docker** tab: the image runs `expanso-edge run` by default and bootstraps the node from the token.

**Option 2: Manual Installation**
Contact [support@expanso.io](mailto:support@expanso.io) for Windows binary download instructions.

**Option 3: WSL2**
Install via Windows Subsystem for Linux (WSL2):
```bash
# Inside WSL2 Ubuntu/Debian
curl -fsSL https://get.expanso.io/edge/install.sh | bash
```
:::

```bash
# Pull Expanso Edge image
docker pull ghcr.io/expanso-io/expanso-edge:latest

# Run Expanso Edge with your bootstrap token
docker run -d \
  --name expanso-edge \
  --restart unless-stopped \
  -e EXPANSO_EDGE_BOOTSTRAP_TOKEN=YOUR_BOOTSTRAP_TOKEN \
  ghcr.io/expanso-io/expanso-edge:latest
```

Get `YOUR_BOOTSTRAP_TOKEN` from the **Keys** tab in [Expanso Cloud](https://cloud.expanso.io). The image runs `expanso-edge run` by default, which bootstraps the node from `EXPANSO_EDGE_BOOTSTRAP_TOKEN` on first start. Without the token the container exits with `edge needs to be bootstrapped`.

### Multiple Nodes with Docker Compose

Run multiple edge nodes on a single host with Docker Compose.

**Step 1: Create `docker-compose.yml`**

```yaml
services:
  edge-1:
    image: ghcr.io/expanso-io/expanso-edge:${VERSION:-latest}
    container_name: expanso-edge-1
    hostname: edge-1
    environment:
      - EXPANSO_EDGE_NAME=edge-1
      - EXPANSO_EDGE_BOOTSTRAP_TOKEN=${EXPANSO_EDGE_BOOTSTRAP_TOKEN}
      - EXPANSO_EDGE_BOOTSTRAP_URL=${EXPANSO_EDGE_BOOTSTRAP_URL}
      - EXPANSO_EDGE_LOG_LEVEL=${EXPANSO_EDGE_LOG_LEVEL:-info}
    volumes:
      - edge-1-data:/var/lib/expanso/edge

  edge-2:
    image: ghcr.io/expanso-io/expanso-edge:${VERSION:-latest}
    container_name: expanso-edge-2
    hostname: edge-2
    environment:
      - EXPANSO_EDGE_NAME=edge-2
      - EXPANSO_EDGE_BOOTSTRAP_TOKEN=${EXPANSO_EDGE_BOOTSTRAP_TOKEN}
      - EXPANSO_EDGE_BOOTSTRAP_URL=${EXPANSO_EDGE_BOOTSTRAP_URL}
      - EXPANSO_EDGE_LOG_LEVEL=${EXPANSO_EDGE_LOG_LEVEL:-info}
    volumes:
      - edge-2-data:/var/lib/expanso/edge

  edge-3:
    image: ghcr.io/expanso-io/expanso-edge:${VERSION:-latest}
    container_name: expanso-edge-3
    hostname: edge-3
    environment:
      - EXPANSO_EDGE_NAME=edge-3
      - EXPANSO_EDGE_BOOTSTRAP_TOKEN=${EXPANSO_EDGE_BOOTSTRAP_TOKEN}
      - EXPANSO_EDGE_BOOTSTRAP_URL=${EXPANSO_EDGE_BOOTSTRAP_URL}
      - EXPANSO_EDGE_LOG_LEVEL=${EXPANSO_EDGE_LOG_LEVEL:-info}
    volumes:
      - edge-3-data:/var/lib/expanso/edge

volumes:
  edge-1-data:
  edge-2-data:
  edge-3-data:
```

**Step 2: Create `.env` file with your bootstrap credentials**

```bash
# Get these values from Expanso Cloud
EXPANSO_EDGE_BOOTSTRAP_TOKEN=your-bootstrap-token
EXPANSO_EDGE_BOOTSTRAP_URL=https://cloud.expanso.io

# Optional settings
EXPANSO_EDGE_LOG_LEVEL=info
VERSION=latest
```

**Step 3: Start the nodes**

```bash
docker compose up -d

# Check status
docker compose ps

# View logs
docker compose logs -f

# Stop nodes (preserves data)
docker compose down

# Stop and remove all data
docker compose down -v
```

Each node has its own persistent volume, so data survives restarts.

**Step 1: Create a Secret for the bootstrap token**

```bash
# Get your bootstrap token from Expanso Cloud, then create a Secret
kubectl create secret generic expanso-bootstrap \
  --from-literal=token=YOUR_BOOTSTRAP_TOKEN
```

**Step 2: Deploy Expanso Edge as a sidecar.**

In this example, we are deploying the edge node as a sidecar for nginx, but this can be replaced with your own app. Additionally, you can run Expanso Edge by itself if that suits your usecase.

```bash
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: shared-logs
              mountPath: /var/log/nginx

        - name: expanso-edge
          image: ghcr.io/expanso-io/expanso-edge:latest
          env:
            - name: EXPANSO_EDGE_BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: expanso-bootstrap
                  key: token
          volumeMounts:
            - name: expanso-cache
              mountPath: /var/lib/expanso
            - name: shared-logs
              mountPath: /var/log/nginx

      volumes:
        - name: expanso-cache
          emptyDir: {}
        - name: shared-logs
          emptyDir: {}
EOF
```

:::tip[Kubernetes Best Practices]
- **Use Secrets for tokens**: Never hardcode bootstrap tokens in manifests. Use Kubernetes Secrets and reference them via `secretKeyRef`.
- **Prefer stdout/stderr**: Kubernetes captures container stdout/stderr logs by default. Emitting logs there integrates with cluster logging drivers.
- **Use a shared volume for file logs**: If your app writes logs to files, mount them on an `emptyDir` (or `hostPath` for host-level logs) and point Expanso at those paths.
:::

---

## Run Expanso Edge

Once installed, start the edge node. This section does not apply to Docker, Docker Compose, or Kubernetes: the container already runs `expanso-edge run` and bootstraps from `EXPANSO_EDGE_BOOTSTRAP_TOKEN`.

First, bootstrap the node with a bootstrap token from your workspace in [Expanso Cloud](https://cloud.expanso.io) (the **Add Node** flow generates this for you):

```bash
expanso-edge bootstrap --token YOUR_BOOTSTRAP_TOKEN
```

Then start the node:

```bash
expanso-edge run
```

Alternatively, pass the token to `run` with `--bootstrap-token YOUR_BOOTSTRAP_TOKEN`, or set the `EXPANSO_EDGE_BOOTSTRAP_TOKEN` environment variable.

### Options

```bash
# Run with custom name (defaults to hostname)
expanso-edge run --name my-edge-node

# Run with an agent config file (agent settings, not pipeline YAML)
expanso-edge run --config /path/to/agent-config.yaml

# Run with verbose logging
expanso-edge run --verbose

# Or set log level
expanso-edge run --log-level debug
```

---

## Install expanso-cli

**expanso-cli** is the command-line interface for interacting with Expanso Cloud and edge nodes via API.
It is optional if you only use the Expanso Cloud web UI, and required for
[Local Mode](/getting-started/local-mode), where it submits jobs to the agent.

**When you need it:**
- Local development with [Local Mode](/getting-started/local-mode)
- CI/CD automation and scripting
- Programmatic job deployment
- API access without writing code

**When you don't need it:**
- Deploying and managing pipelines entirely from the Expanso Cloud web UI

Note that deploying a pipeline always goes through a job submission, so running
pipelines from the command line needs `expanso-cli` even on a standalone agent.
`expanso-edge run --config` loads agent settings and does not run a pipeline.

**Installation:**

```bash
curl -fsSL https://get.expanso.io/cli/install.sh | bash
```

**Verify installation** (once installed):
```bash
expanso-cli version
```

---

## Verify What You Downloaded

Binaries and container images ship with signed build provenance, so you can confirm an artifact came from Expanso's release pipeline and hasn't been modified since. Verification needs no account and no access to Expanso's source repository:

```bash
gh attestation verify expanso-edge_${VERSION}_linux_amd64.tar.gz \
  --bundle expanso-edge_${VERSION}_provenance.intoto.jsonl \
  --owner expanso-io \
  --signer-workflow expanso-io/expanso/.github/workflows/_release.yml
```

See **[Build Provenance](/references/build-provenance)** for the full procedure, including container images, air-gapped verification, and which releases are covered.

---

## What's Next?

- **[Quick Start](/getting-started/quickstart/)**: Build your first pipeline
- **[Core Concepts](/getting-started/concepts/)**: Understand how Expanso works
- **[Components](/components)**: Explore available inputs, processors, and outputs
- **[Local Mode](/getting-started/local-mode)**: Use expanso-cli for local development
- **[Build Provenance](/references/build-provenance)**: Verify the artifacts you installed

## Local Mode

# Local Mode

Run Expanso Edge as a standalone API server for rapid pipeline development and testing, with no workspace or cloud connection required.

## What is Local Mode?

Local mode runs Expanso Edge with a built-in API server that accepts job submissions directly via the CLI. Perfect for:

- **First-time users** - Get started without infrastructure
- **Pipeline development** - Fast iteration on pipeline configurations
- **Debugging** - Test pipelines locally before deployment
- **CI/CD** - Automated pipeline testing

**What you get:**
- ✅ Full pipeline processing capabilities
- ✅ REST API for job submission
- ✅ CLI integration for job management
- ✅ Local state storage
- ✅ Zero configuration needed

**What you don't get:**
- ❌ Multi-node coordination
- ❌ Centralized orchestration
- ❌ Job scheduling and assignment
- ❌ Fleet management
- ❌ Web UI for monitoring

---

## Get Started

Follow these guides to start using local mode:

### [Quick Start](/getting-started/local-mode/quick-start)
Run a hello-world pipeline on your machine and see its first record.

### [CLI Commands](/getting-started/local-mode/cli-commands)
Learn how to deploy, manage, and monitor jobs using the Expanso CLI.

### [Examples](/getting-started/local-mode/examples)
Practical examples for file processing, data transformation, and HTTP webhooks.

### [Troubleshooting](/getting-started/local-mode/troubleshooting)
Solutions to common issues like port conflicts, job format errors, and connection problems.

---

## Quick Command Reference

```bash
# Start edge in local mode
expanso-edge run --local

# In another terminal: keep CLI settings in this folder
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"

# Deploy a pipeline to the local edge
expanso-cli job deploy pipeline.yaml \
  --endpoint http://localhost:9010

# List running jobs
expanso-cli job list --endpoint http://localhost:9010

# Stop a job
expanso-cli job stop <job-id> --endpoint http://localhost:9010
```

---

## When to Use Local Mode

**Use local mode when:**
- Learning Expanso for the first time
- Developing new pipeline configurations
- Testing pipelines before production deployment
- Running automated pipeline tests in CI/CD
- Debugging pipeline behavior locally

**Use orchestrated mode when:**
- Deploying to production infrastructure
- Managing multiple edge nodes
- Needing centralized job scheduling
- Requiring web UI for monitoring
- Setting up fleet management

---

## Next Steps

- **[Quick Start Guide](/getting-started/local-mode/quick-start)** - Your first local pipeline
- **[Deploy to Production](/getting-started/quickstart/)** - Move to Expanso Cloud
- **[Testing & Debugging](/getting-started/testing-debugging)** - Advanced techniques

## CLI Commands

# CLI Commands

Deploy, monitor, and manage jobs running on your local edge node.

## Configure CLI for Local Edge

In each terminal you use for this project, run this from the project folder
so the CLI keeps its settings there instead of in your home directory:

```bash
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
```

Then pass `--endpoint http://localhost:9010` on every command, as below. The
per-folder settings and the explicit endpoint keep the CLI from using a saved
profile or an Expanso Cloud workspace you may have set up elsewhere.

---

## Job Commands

### Deploy a Job

```bash
expanso-cli job deploy my-pipeline.yaml \
  --endpoint http://localhost:9010
```

### List Running Jobs

```bash
expanso-cli job list --endpoint http://localhost:9010
```

### Get Job Status

```bash
expanso-cli job describe <job-id> \
  --endpoint http://localhost:9010
```

### Stop a Job

```bash
expanso-cli job stop <job-id> --endpoint http://localhost:9010
```

### View Job Executions

```bash
expanso-cli job executions <job-id> \
  --endpoint http://localhost:9010
```

---

## Configuration Options

### Custom API Port

Local mode listens on port `9010` by default. To change it:

```bash
expanso-edge run --local --api-listen :8080
```

Then pass that port to the CLI instead:

```bash
expanso-cli job list --endpoint http://localhost:8080
```

### Data Directory

Specify a custom location for local state:

```bash
expanso-edge run --local --data-dir ./my-data
```

### Debug Logging

Enable detailed execution logs:

```bash
expanso-edge run --local --log-level debug
```

---

## Next Steps

- **[Examples](/getting-started/local-mode/examples)** - Practical pipeline examples
- **[Troubleshooting](/getting-started/local-mode/troubleshooting)** - Fix common issues

## Examples

# Local Mode Examples

Copy-paste examples to get started with common pipeline patterns.

:::tip[Looking for More?]
These examples use just a few of the available components. Browse **[all inputs, processors, and outputs](/components)** for the full list.
:::

---

## File Processing

Read log files and filter for errors only.

Create `logs-job.yaml`:

```yaml
name: log-processor
description: Process log files and extract errors
type: pipeline
config:
  input:
    file:
      paths: ["/var/log/app/*.log"]
      codec: lines

  pipeline:
    processors:
      - mapping: |
          root = this.parse_json()
          root = if this.level != "ERROR" { deleted() }

  output:
    file:
      path: ./errors.log
      codec: lines
```

Deploy it from the same folder:

```bash
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job deploy logs-job.yaml \
  --endpoint http://localhost:9010
```

---

## Data Transformation

Convert a CSV file to JSON format.

Create `csv-transform-job.yaml`:

```yaml
name: csv-to-json
description: Transform CSV users file to JSON
type: pipeline
config:
  input:
    file:
      paths: ["./users.csv"]
      codec: lines

  pipeline:
    processors:
      - mapping: |
          root = if @message_number == 0 { deleted() }

          let parts = this.split(",")
          root = {
            "name": $parts.index(0),
            "email": $parts.index(1),
            "age": $parts.index(2).number()
          }

  output:
    file:
      path: ./users.json
      codec: lines
```

---

## HTTP Webhook Receiver

Listen for incoming webhook requests and process them.

Create `http-receiver-job.yaml`:

```yaml
name: webhook-receiver
description: Receive and process webhook data
type: pipeline
config:
  input:
    http_server:
      address: "0.0.0.0:8081"
      path: /webhook

  pipeline:
    processors:
      - mapping: |
          root = this.parse_json()
          root.processed_at = now()

  output:
    stdout:
      codec: lines
```

Test it:

```bash
# Terminal 1: Start edge
expanso-edge run --local

# Terminal 2: Deploy job
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job deploy http-receiver-job.yaml \
  --endpoint http://localhost:9010

# Terminal 3: Send test data
curl -X POST http://localhost:8081/webhook \
  -H "Content-Type: application/json" \
  -d '{"event":"test","data":"hello"}'
```

---

## Common Workflows

### Development Iteration

```bash
# Terminal 1: Start edge once
expanso-edge run --local -v

# Terminal 2: Iterate on pipelines
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job deploy v1-pipeline.yaml \
  --endpoint http://localhost:9010
# ... test, update file ...
expanso-cli job deploy v2-pipeline.yaml \
  --endpoint http://localhost:9010
```

### Multiple Pipelines

Run several pipelines at once:

```bash
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job deploy pipeline-a.yaml \
  --endpoint http://localhost:9010
expanso-cli job deploy pipeline-b.yaml \
  --endpoint http://localhost:9010
expanso-cli job list --endpoint http://localhost:9010
```

---

## Next Steps

- **[Troubleshooting](/getting-started/local-mode/troubleshooting)** - Fix common issues
- **[Components](/components)** - Browse all inputs, processors, and outputs

## Quick Start

# Local Mode Quick Start

Get a first record from a local pipeline in under a minute, with no cloud
connection required.

## Prerequisites

- Expanso Edge installed ([Installation Guide](/getting-started/installation))
- `expanso-cli` installed ([Install expanso-cli](/getting-started/installation#install-expanso-cli)).
  Local mode needs it to submit jobs to the agent.
- Terminal access

:::tip[Want a faster first result?]
[Try It Without an Account](/getting-started/try-locally) runs a log-filtering
pipeline that you can send your own log lines to.
:::

---

## Step 1: Start Edge in Local Mode

Run this from an empty working folder:

```bash
expanso-edge run --local --data-dir ./data
```

Among the startup lines (timestamps trimmed) you'll see:

```
INF API server listening addr=127.0.0.1:9010 component=api_server
INF Expanso Edge Service started successfully component=edge_service
```

The edge node is now running and ready to accept pipeline jobs on port `9010`.
`--data-dir ./data` keeps its job store, execution state, and pipeline logs
in `./data`. Without it, the agent uses `~/.expanso/edge`.

---

## Step 2: Create a Simple Job

Create `hello-job.yaml`:

```yaml
name: hello-world
description: Simple hello world pipeline
type: pipeline
config:
  input:
    generate:
      count: 5
      interval: "1s"
      mapping: |
        root.message = "Hello from pipeline!"
        root.timestamp = now()
        root.count = count("messages")

  pipeline:
    processors: []

  output:
    stdout:
      codec: lines
```

---

## Step 3: Validate the Job

In a new terminal, from the same folder, give the CLI a settings folder of
its own, then validate the job:

```bash
# Keep CLI settings in this folder, not your home directory
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"

expanso-edge validate hello-job.yaml
```

This checks the file offline without running it. You should see:

```
[OK] hello-job.yaml: valid
```

---

## Step 4: Submit the Pipeline

In the same terminal, submit the pipeline to the local agent:

```bash
expanso-cli job deploy hello-job.yaml \
  --endpoint http://localhost:9010
```

You should see:

```
Job 'hello-world' created successfully in namespace ''
```

The per-folder `EXPANSO_CLI_CONFIG_DIR` and the explicit `--endpoint` keep the
CLI from using a saved profile or an Expanso Cloud workspace you may have set
up elsewhere; use both on every `expanso-cli` command against the local agent.

---

## Step 5: Monitor the Job

Watch the output in the terminal where `expanso-edge` is running. You'll see:

```json
{"count":1,"message":"Hello from pipeline!","timestamp":"..."}
{"count":2,"message":"Hello from pipeline!","timestamp":"..."}
{"count":3,"message":"Hello from pipeline!","timestamp":"..."}
{"count":4,"message":"Hello from pipeline!","timestamp":"..."}
{"count":5,"message":"Hello from pipeline!","timestamp":"..."}
```

Each `timestamp` is the local time the message was generated. After five
messages the job finishes, and
`expanso-cli job list --endpoint http://localhost:9010` shows it as
`completed`.

🎉 **Success!** You've run your first pipeline in local mode.

---

## What's Next?

Now that you have local mode running:

- **[Local Mode Overview](/getting-started/local-mode)** - Full local mode documentation
- **[Components](/components)** - Explore 200+ inputs, processors, and outputs
- **[Bloblang Guide](/guides/bloblang)** - Learn data transformations

### Common Next Steps

1. **Try More Examples** - Explore file processing, HTTP webhooks, and data transformations
2. **Build Your Own Pipeline** - Modify the example to process your data
3. **Test Before Production** - Use local mode to validate pipelines before deployment
4. **Deploy to Production** - Move to [Expanso Cloud](/getting-started/quickstart/) when ready

---

## Quick Tips

**Fast Feedback:** Local mode starts instantly—no bootstrap, no credentials needed.

**Per-folder CLI settings:** Set `EXPANSO_CLI_CONFIG_DIR` in each terminal
you use for this project, and pass `--endpoint` on every command, for example:
```bash
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job list --endpoint http://localhost:9010
```

**Clean State:** Stop edge (Ctrl+C), remove the `./data` directory you passed
to `--data-dir` (and `./.expanso-cli`, if present), and restart fresh.

**Debug Logging:** See what's happening inside:
```bash
expanso-edge run --local --data-dir ./data --log-level debug
```

## Troubleshooting

# Troubleshooting

Solutions to common issues when running local mode.

## Job Format Errors

### "job type cannot be empty"

Jobs submitted via CLI must use the wrapper format:

```yaml
name: my-job
type: pipeline
config:
  input: { generate: { interval: 5s, mapping: 'root.hello = "world"' } }
  output: { stdout: {} }
```

❌ **Won't work** - missing wrapper:
```yaml
input:
  generate: { ... }
```

### "pipeline 'input' component is required" / "pipeline 'output' component is required"

Every job config needs an `input` and an `output`. The `pipeline` section (processors) is optional; add whichever of `input` or `output` is missing:

```yaml
name: my-job
type: pipeline
config:
  input: { generate: { interval: 5s, mapping: 'root.hello = "world"' } }
  output: { stdout: {} }
```

---

## Connection Issues

### CLI can't connect to the local edge

The CLI reports `could not connect to localhost:9010: connection refused` when it targets the wrong port. The default is `9010`, not `8080`. Pass it explicitly:

```bash
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job list --endpoint http://localhost:9010
```

If you started edge with a custom port:

```bash
expanso-edge run --local --api-listen :8080

# In another terminal, from the same folder
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job list --endpoint http://localhost:8080
```

### Port Already in Use

**Error:** `Error: listen tcp :9010: bind: address already in use`

Use a different port:

```bash
expanso-edge run --local --api-listen :8080

# In another terminal, from the same folder
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job list --endpoint http://localhost:8080
```

### CLI Can't Find Edge

**Error:** `could not connect to localhost:9010: connection refused` (with the hint: No orchestrator is reachable at localhost:9010. Ensure one is running.)

1. Verify edge is running—look for the "API server listening" line
2. Check the port matches your `--api-listen` setting
3. Ensure your firewall allows localhost connections

---

## Job Issues

### Job Not Running

1. **Check job status:**
   ```bash
   export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
   expanso-cli job list --endpoint http://localhost:9010
   expanso-cli job describe <job-id> \
     --endpoint http://localhost:9010
   ```

2. **Enable debug logging:**
   ```bash
   expanso-edge run --local --log-level debug
   ```

### Old Jobs Still Appearing

Clean the state directory:

```bash
# Stop edge (Ctrl+C)
rm -rf ./data
expanso-edge run --local
```

---

## Next Steps

- **[CLI Commands](/getting-started/local-mode/cli-commands)** - Full command reference
- **[Local Mode Overview](/getting-started/local-mode)** - Return to main guide

## Next Steps

# Next Steps

You've completed the getting started guides! Here's where to go next.

## Deepen Your Knowledge

### Master Data Transformations

**[Bloblang Guide](/guides/bloblang)** - Learn the powerful transformation language

Topics covered:
- Functions and methods
- Conditional logic
- Working with JSON, arrays, objects
- Advanced patterns

---

### Explore Components

**[Component Catalog](/components)** - Browse all 200+ inputs, processors, and outputs

Find components for:
- Your data sources (databases, queues, APIs)
- Your destinations (cloud storage, analytics platforms)
- Your transformation needs (parsing, enrichment, filtering)

---

### Study Real Examples

**[Use Cases](/use-cases)** - See how others use Expanso

- Log processing and aggregation
- IoT data collection and filtering
- Real-time analytics at the edge
- Security and compliance patterns

---

## Build Production Pipelines

### Advanced Pipeline Patterns

**[Pipeline Guides](/guides/pipelines/about)** - Deep dive into pipeline features

Learn about:
- Error handling and retry logic
- Batching and windowing
- Resource management
- Metadata and interpolation

---

### Deploy at Scale

**[Infrastructure Guides](/guides/infrastructure)** - Production deployment patterns

Topics:
- Kubernetes deployments
- Docker Compose setups
- Fleet management
- High availability

---

### Integrate with Cloud Services

**[Cloud Integration Guides](/guides/cloud/aws)** - Connect to AWS, GCP, Azure

Learn how to:
- Authenticate with cloud providers
- Use cloud-native components
- Optimize for cost and performance

---

## Operations

### Manage Secrets

**[Secrets Management](/guides/secrets-management)** - Handle credentials securely

- Bootstrap secrets
- External secret providers
- Best practices

---

## Get Help

- **Documentation**: [docs.expanso.io](https://docs.expanso.io)
- **Support**: support@expanso.io
- **API Reference**: [/references/api](/references/api)
- **CLI Reference**: [/references/cli](/references/cli)

---

## Share Your Feedback

We'd love to hear about:
- What you're building with Expanso
- Features you'd like to see
- Documentation improvements

Reach out at support@expanso.io

## Cloud Quickstart

# Cloud Quickstart

Get your first data pipeline running with Expanso Cloud in under 10 minutes.

## What You'll Do

Follow these 4 steps to deploy your first edge data pipeline:

1. **[Sign Up](/getting-started/quickstart/sign-up)** - Create your Expanso Cloud account
2. **[Deploy Node](/getting-started/quickstart/deploy-node)** - Install and bootstrap an edge node
3. **[Build Pipeline](/getting-started/quickstart/first-pipeline)** - Create your first data pipeline
4. **[View Results](/getting-started/quickstart/view-results)** - Monitor your pipeline in action

**Time required**: 5-10 minutes

---

## Prerequisites

- A computer running Linux, macOS, or Windows
- Internet connection
- Terminal access
- Valid email address

---

## Alternative Paths

**Want to develop locally first?**
Check out the **[Local Development](/getting-started/local-mode)** path to build and test pipelines on your machine without any cloud setup.

**Want to understand the concepts first?**
Visit **[Core Concepts](/getting-started/concepts/)** to learn how Expanso works before diving into hands-on tutorials.

---

## Ready to Start?

👉 **[Begin with Step 1: Sign Up](/getting-started/quickstart/sign-up)**

## Deploy Your First Node

# Deploy Your First Node

An edge **node** runs on your infrastructure and executes data pipelines. Let's deploy one to your workspace.

## Deploy Your Node

There are two ways to deploy a node — both register it with your workspace using a bootstrap token. Pick whichever you prefer.

### Option 1: Generate an install command (Nodes tab)

Let Expanso Cloud build a ready-to-run install command for you:

1. Open Expanso Cloud and go to the **Nodes** tab
2. Click **Add Node**
3. Choose how long the generated token should stay valid
4. Copy the generated install command and run it on your machine

The command installs Expanso Edge and bootstraps the node. Then [start the node](#start-the-node). If you chose the Docker command, the container is already running, so skip ahead to [Verify Connection](#verify-connection).

### Option 2: Create a bootstrap token (Keys tab)

Prefer to install Expanso Edge yourself? Create a token, then run the commands below.

1. Open Expanso Cloud and go to the **Keys** tab
2. Create a bootstrap token, choosing how long it stays valid
3. Copy the token, then run these commands on Linux or macOS:

```bash
# Install Expanso Edge
curl -fsSL https://get.expanso.io/edge/install.sh | bash

# Bootstrap the node with your token
expanso-edge bootstrap --token YOUR_BOOTSTRAP_TOKEN
```

Replace `YOUR_BOOTSTRAP_TOKEN` with the token you copied.

For Docker, Docker Compose, Kubernetes, Windows, and Android, follow the [Installation guide](/getting-started/installation#install-expanso-edge) with the same token instead.

### Start the node

Bootstrapping stores the node's credentials but does not start it. Start the node and leave it running:

```bash
expanso-edge run
```

---

## Verify Connection

1. Return to Expanso Cloud
2. Navigate to the **Nodes** tab in your workspace
3. Your node should appear (usually named after your hostname)
4. Status shows **Connected** (green)

Your node is connected and ready to run pipelines.

---

## What's Next?

Next step: **[Build Your First Pipeline](/getting-started/quickstart/first-pipeline)**

---

## Troubleshooting

**Node not appearing?**
- Wait 30 seconds and refresh the page
- Confirm `expanso-edge run` is still running and check its output for errors (for Docker, `docker logs <container>`)
- Verify network connectivity to cloud.expanso.io

**Bootstrap failed?**
- Ensure token is correct (no extra spaces)
- Generate a new token if expired
- Check firewall allows outbound HTTPS

**Node shows offline?**
- The node may have stopped -- restart it with `expanso-edge run`
- Check system resources (CPU, memory, disk)

For system requirements and all installation options, see the [Installation guide](/getting-started/installation).

## Build Your First Pipeline

# Build Your First Pipeline

Create a simple log processing pipeline using the Expanso Cloud pipeline builder.

## What You'll Build

A pipeline that:
1. Reads log files from your node
2. Filters for ERROR-level logs
3. Writes to STDOUT

---

## Step 1: Create a New Pipeline

1. In your workspace, go to the **Pipelines** tab
2. Click **New Pipeline**
3. On **"How do you want to start?"**, choose **Start from Scratch**

This opens an empty pipeline in the visual builder.

---

## Step 2: Build the Pipeline

The builder opens in the **Design** view. Add and configure three components using their forms.

**Input: File**
1. Click **+ Add Input** and select **File**
2. In the component form, add the path `/var/log/*.log`

This reads all `.log` files from `/var/log/`.

**Processor: Mapping** (uses Bloblang)
1. Click **Add Processor** and select **Mapping**
2. In the mapping field, enter:
   ```
   # Only keep ERROR logs
   root = if this.contains("ERROR") {
     this
   } else {
     deleted()
   }
   ```

**Output: STDOUT**
1. Click **+ Add Output** and select **STDOUT**
2. In the component form, set the codec to `lines`

Your pipeline now shows `File → Filter → STDOUT`. Click **Next**.

---

## Step 3: Pipeline Details and Deploy

On the **Pipeline Details** step:

1. Enter a **Name**: `error-log-filter` (lowercase letters, numbers, and hyphens)
2. Under **Node Selector**, choose which nodes run the pipeline (leave it empty to target every node in the workspace, or match specific labels)
3. Click **Create Pipeline**

The pipeline deploys to the selected nodes within seconds.

---

## What's Next?

👉 **[View Results & Monitor](/getting-started/quickstart/view-results)**

---

## Tips

**Visual flow**: Your pipeline shows: `File → Filter → STDOUT`

**Edit anytime**: Click the pipeline to edit and redeploy

**Multiple outputs**: Add more outputs to send data to multiple destinations

**Learn Bloblang**: See [Bloblang Guide](/guides/bloblang) for advanced transformations

## Sign Up for Expanso Cloud

# Sign Up for Expanso Cloud

Create your free Expanso Cloud account to start deploying edge data pipelines.

## Sign In

1. Visit [cloud.expanso.io/login](https://cloud.expanso.io/login)
2. Enter your email and click **Send magic link**, or choose **Continue with Google**, **GitHub**, or **LinkedIn**
3. If you used email, open the message from Expanso and click the magic link
4. You are signed in and land on the Expanso Cloud dashboard

There is no password to create and no separate verification step: clicking the magic link both verifies your email and signs you in.

---

## First Login

The first time you sign in:

1. A **personal organization** is created for you automatically, with a **default workspace** already provisioned
2. Your account is ready to start deploying right away

---

## What's Next?

Your account and default workspace are ready:

👉 **[Deploy Your First Node](/getting-started/quickstart/deploy-node)**

---

## Troubleshooting

**Didn't receive the magic link?**
- Check your spam folder
- Click **Resend link** (available 60 seconds after the first request)
- Contact support@expanso.io if issues persist

**Link not working?**
- Magic links expire after 5 minutes and can only be used once; request a new one if yours expired
- Make sure you opened the most recent link
- Ensure you entered the correct email address

## View Results & Monitor

# View Results & Monitor

See your pipeline in action and monitor its performance.

## View Pipeline Status

1. Go to **Pipelines** tab in your workspace
2. Find your `error-log-filter` pipeline
3. Status shows **Running** (green)
4. Click the pipeline name to see details

---

## Monitor the Pipeline

Open the pipeline's **Monitoring** tab to see its charts:

- **Input Data Rate** and **Output Data Rate**: bytes flowing in and out
- **Pipeline Orchestration Errors**: orchestration errors over the selected window

Choose the window with the time range picker, and use the refresh button to update the charts, which do not update on their own.

For per component metrics, open the pipeline's **Overview** and click a component in the graph to see its **Latency Percentiles**.

---

## View the Output

The pipeline writes to STDOUT. View that output from the pipeline's **Logs** tab, which streams logs live (requires the node and workspace on v2.1.18 or newer).

You can also stream the same logs from the CLI:

```bash
expanso-cli job logs error-log-filter
```

You'll see the filtered ERROR log lines being output.

---

## Understanding What Happened

Your pipeline just:
1. ✅ Read log files from `/var/log/*.log`
2. ✅ Filtered to keep only ERROR lines
3. ✅ Output results to stdout
4. ✅ Sent metrics to Expanso Cloud

All processing happened **at the edge** on your node, not in the cloud!

---

## What's Next?

Congratulations! You've completed the Cloud Quickstart. You now know how to:
- ✅ Create an Expanso Cloud account
- ✅ Deploy an edge node
- ✅ Build and deploy a pipeline
- ✅ Monitor pipeline performance

### Continue Learning

- **[Plan Your Next Steps](/getting-started/next-steps)** - Choose guides for production pipelines, integrations, and monitoring

- **[Core Concepts](/getting-started/concepts/)** - Understand how Expanso works in depth
- **[Components](/components)** - Explore 200+ inputs, processors, and outputs
- **[Use Cases](/use-cases)** - See real-world pipeline examples
- **[Bloblang Guide](/guides/bloblang)** - Master data transformations

### Build More

- Connect to real data sources (Kafka, databases, APIs)
- Transform and enrich data with Bloblang
- Send to cloud destinations (S3, Elasticsearch, etc.)
- Deploy multiple nodes across your infrastructure
- Set up monitoring and alerts

---

## Need Help?

- **Documentation**: Browse [docs.expanso.io](https://docs.expanso.io)
- **Support**: Email support@expanso.io
- **Community**: Join our community forum

## Testing & Debugging

# Testing & Debugging

Learn how to validate configurations, add debugging output, and troubleshoot common issues when building pipelines.

## Quick Reference

```bash
# Validate a pipeline config offline (no control plane needed)
expanso-edge validate my-pipeline.yaml

# Validate a job spec against the control plane
expanso-cli job validate my-job.yaml

# Validate a job spec without contacting the control plane
expanso-cli job validate my-job.yaml --offline

# Deploy, then stream the pipeline's log
# (records appear only via a log processor)
expanso-cli job deploy my-job.yaml
expanso-cli job logs my-job

# Test with limited data
# (add count: 10 to your generate input)

# Output to terminal for inspection
# (use stdout output in your config)
```

Throughout this page, `my-pipeline.yaml` is a bare **pipeline config**
(`input`/`pipeline`/`output` at the root) and `my-job.yaml` is the **job spec**
that wraps it — `name` and `type: pipeline` at the top level with that config
nested under `config:`. See
[Step 1](/getting-started/first-pipeline/hello-world#run-it) for the wrapper.
`expanso-edge validate` takes the pipeline config; every `expanso-cli job`
command takes the job spec.

---

## Editor Setup

Get autocomplete and validation while writing pipeline configs - makes authoring faster and catches errors before you run anything.

Expanso provides a JSON Schema for pipeline YAML files. Point your editor at it and you'll get autocomplete for component names, validation for config fields, and inline docs without leaving your editor.

**What you'll get:**
- Browse available components as you type (inputs, processors, outputs)
- Catch typos and missing fields immediately
- See component docs on hover
- Write configs faster with fewer errors

### Configure VS Code

Add this to `.vscode/settings.json` in your workspace (or global settings):

```json
{
  "yaml.schemas": {
    "https://docs.expanso.io/schemas/pipeline.schema.json": [
      "**/*.pipeline.yaml",
      "**/*pipeline*.yaml"
    ]
  }
}
```

This maps the Expanso pipeline schema to any YAML files matching these patterns.

**Per-workspace setup** (recommended):

1. Create `.vscode/settings.json` in your project directory
2. Add the schema configuration above
3. Commit it to version control so your team gets autocomplete automatically

**Global setup** (applies to all projects):

1. Open VS Code settings (Cmd+, or Ctrl+,)
2. Search for "yaml.schemas"
3. Click "Edit in settings.json"
4. Add the schema configuration

### Configure IntelliJ IDEA / PyCharm

1. Go to **Preferences** → **Languages & Frameworks** → **Schemas and DTDs** → **JSON Schema Mappings**
2. Click **+** to add a new schema
3. Set:
   - **Name**: Expanso Pipeline Schema
   - **Schema file or URL**: `https://docs.expanso.io/schemas/pipeline.schema.json`
   - **Schema version**: JSON Schema version 7
4. Add file path patterns:
   - `**/*.pipeline.yaml`
   - `**/*pipeline*.yaml`
5. Click **OK**

### Configure Other Editors

Most editors with YAML support can use JSON Schema for validation.

**Neovim** (with yaml-language-server):

Add to your LSP config or `.luarc.json`:

```json
{
  "yaml.schemas": {
    "https://docs.expanso.io/schemas/pipeline.schema.json": "**/*.pipeline.yaml"
  }
}
```

**Sublime Text** (with LSP-yaml):

Add to LSP-yaml settings:

```json
{
  "settings": {
    "yaml.schemas": {
      "https://docs.expanso.io/schemas/pipeline.schema.json": "**/*.pipeline.yaml"
    }
  }
}
```

**General approach**: Look for YAML language server or JSON Schema support in your editor's docs. Configure it to associate `https://docs.expanso.io/schemas/pipeline.schema.json` with your pipeline YAML files.

### What You'll See

Once configured:

**Autocomplete for components**:

Start typing under `input:`, `processors:`, or `output:` and you'll get suggestions for all available components (kafka, http_server, generate, mapping, etc.).

**Parameter completion**:

When configuring a component, autocomplete shows valid configuration fields. For example, typing under `kafka:` suggests `addresses`, `topics`, `consumer_group`, and other Kafka-specific options.

**Inline validation**:

Red squiggly lines appear immediately when you:
- Misspell a component name
- Forget required fields
- Use invalid configuration keys
- Have incorrect YAML indentation

**Hover documentation**:

Hover over any component or field to see its description and usage notes without switching to the docs.

### File Naming

The schema works best with these naming patterns:

- `*.pipeline.yaml` - Standard pipeline configs
- `log-processor.pipeline.yaml` - Descriptive names
- `my-pipeline.yaml` - Also works if you add the pattern to your schema config

You can customize the file patterns in your editor's schema configuration to match your team's naming conventions.

### Limitations

The JSON Schema provides syntax validation and autocomplete, but it doesn't validate:

- **Bloblang expressions**: Your mapping logic syntax isn't checked
- **Runtime values**: Environment variables like `${VAR}` aren't validated
- **Component availability**: All components show in autocomplete, but some may not be available in your Expanso Edge version

Always validate before you run the pipeline:

```bash
expanso-edge validate my-pipeline.yaml
```

This catches issues the schema can't detect, like invalid Bloblang syntax or a field the component doesn't accept.

### Tips

**Use descriptive file names**: `log-processor.pipeline.yaml` is clearer than `pipeline1.yaml`.

**Commit your workspace settings**: Share `.vscode/settings.json` with your team so everyone gets autocomplete automatically.

**Validate often**: Run `expanso-edge validate` to catch issues the schema misses. It needs no server, so it's cheap to run on every save.

---

## Validate Before Running

Always validate your pipeline configuration before running it. This catches syntax errors and configuration mistakes early.

Two commands do this, and they check different things:

| Command | Needs a server | What it checks |
|---------|----------------|----------------|
| `expanso-edge validate` | No | Pipeline config in depth: structure, component fields, Bloblang mappings |
| `expanso-cli job validate --offline` | No | Job spec syntax and structure |
| `expanso-cli job validate` | Yes | Job spec plus comprehensive checks against the control plane |

Start with `expanso-edge validate` while writing the config, then validate the job spec before you deploy.

### Offline Validation on the Node

`expanso-edge validate` is the deepest check you can run without a server. It runs entirely on the machine you invoke it on — no control plane connection, no node credentials, and no edge configuration required — which makes it useful both as a pre-flight check on an edge node and as a CI step.

```bash
expanso-edge validate my-pipeline.yaml
```

```
[OK] my-pipeline.yaml: valid
```

**What it checks:**
- ✅ Valid YAML (or JSON)
- ✅ Config structure - unknown fields, wrong types, unsupported top-level sections
- ✅ Component names exist and accept the fields you set
- ✅ Bloblang mapping syntax

**What it does not check:**
- ❌ Job-level fields - `name`, `type`, `namespace`, selectors, rollout, restart policy
- ❌ Control plane admission - namespace existence, node matching, server dry-run

A config that passes `expanso-edge validate` can still be rejected when you deploy it, so validate the job spec as well before submitting.

#### Reading the Output

Failures list one line per problem, with a position in the file where one is known:

```
[FAIL] my-pipeline.yaml: 2 error(s)
  - (line 7) bloblang syntax error in input.generate.mapping. Expected method or field path (unexpected end of expression)
  - (line 2, col 3) [input.generate]. Unknown field 'bogus_field' in generate component. Check the documentation for valid fields in the 'generate' component.
  hint: Check your Bloblang mapping syntax and pipeline configuration structure.
```

Each error names the path within the config (`input.generate.mapping`), so you can jump straight to the offending key.

#### Input Sources

Validate files, piped input, or several configs at once:

```bash
# One or more files
expanso-edge validate pipeline.yaml

# Several files - each is reported separately
expanso-edge validate pipelines/*.yaml

# Piped input (reads stdin when given "-" or no file at all)
cat pipeline.yaml | expanso-edge validate -
```

A single file or stream can hold multiple YAML documents separated by `---`. Each document is validated independently and labeled with its position:

```
[OK] pipelines.yaml (doc 1): valid
[FAIL] pipelines.yaml (doc 2): 1 error(s)
  - (line 4) bloblang syntax error in input.generate.mapping. Expected query (unexpected end of expression)
  hint: Check your Bloblang mapping syntax. See documentation for valid expressions.
```

#### Job Specs

You can pass a full job spec instead of a bare config, as a convenient way to check the config it carries. The spec itself is only checked for parsing - it must be valid YAML or JSON that reads as a job - and then the embedded `config` block gets the same full validation a bare config does, with reported positions pointing into that embedded config. Job-level fields are not checked; use `expanso-cli job validate` for those.

```bash
expanso-edge validate hello-world-job.yaml
```

Two behaviors are worth knowing:

- A pipeline job with a missing or misspelled `config` key is an error, not a pass:

  ```
  [FAIL] hello-world-job.yaml: 1 error(s)
    - pipeline job has no config to validate (is the 'config' key present and correctly spelled?)
  ```

- Job types that carry no pipeline config pass with a note:

  ```
  [OK] node-check.yaml: valid (query job; no pipeline config to validate)
  ```

#### Exit Codes and Scripting

The command exits `0` only when every document in every input is valid, and non-zero if anything is invalid, empty, or unreadable - so it composes with the rest of your shell without extra parsing:

```bash
expanso-edge validate pipelines/*.yaml
```

Add `--output json` (or `-o json`) for machine-readable results. Validation results go to stdout and the failure summary goes to stderr, so you can pipe straight into `jq`:

```bash
expanso-edge validate pipelines/*.yaml --output json | jq -r '.[] | select(.valid == false) | .source'
```

```json
[
  {
    "source": "bad.yaml",
    "valid": false,
    "kind": "config",
    "hint": "Check your Bloblang mapping syntax and pipeline configuration structure.",
    "errors": [
      {
        "kind": "bloblang",
        "path": "input.generate.mapping",
        "message": "Expected method or field path (unexpected end of expression)",
        "line": 7
      },
      {
        "kind": "schema",
        "path": "input.generate",
        "message": "Unknown field 'bogus_field' in generate component",
        "suggestion": "Check the documentation for valid fields in the 'generate' component.",
        "line": 2,
        "column": 3,
        "component": "generate"
      }
    ]
  }
]
```

Each result carries `source`, `valid`, and `kind` (`config`, `pipeline-job`, `job`, or `error`), plus `note` and, when invalid, `hint` and `errors`. Each error carries `kind` (`schema`, `bloblang`, `policy`, or `yaml`), `message`, and - where known - `path`, `suggestion`, `line`, `column`, `component`, and `label`.

:::note[Mappings that read the local machine]
Validation happens where the pipeline would run, so Bloblang functions that read host state are resolved against the machine running the command. A literal `file("/etc/expanso/token")` in a mapping is read at validation time and reports an error if that path is missing locally - which is what you want on the target node, but means a config can fail on a CI runner and pass on the node (or the reverse). Environment interpolation like `${VAR}` in config fields is not resolved during validation.
:::

### Validating Job Specs with the CLI

`expanso-cli job validate` checks the job spec itself - the fields `expanso-edge validate` deliberately skips.

Client-side only, no server needed:

```bash
expanso-cli job validate my-job.yaml --offline
```

**Example output (success, offline):**

```
✓ Client-side validation passed
Note: Server-side validation skipped (--offline mode)
```

**Example output (error):**

```
Error: pipeline 'input' component is required
```

For comprehensive validation against your Expanso Cloud environment, drop `--offline`:

```bash
expanso-cli job validate my-job.yaml
```

This performs both client-side and server-side checks (requires connection to Expanso Cloud).

---

## Add Debug Output

When developing pipelines, add `log` processors at each stage to see what's happening.

### Debug at Each Stage

```yaml
input:
  file:
    paths: [./data.json]
    codec: lines

pipeline:
  processors:
    # First transformation
    - mapping: |
        root = this.parse_json()
    
    # Debug: log what we parsed
    - log:
        level: INFO
        message: '${! content() }'
    
    # Second transformation
    - mapping: |
        root.processed = true
        root.timestamp = now()

output:
  stdout:
    codec: lines
```

### Use Labels for Clarity

Add labels to track which stage produced output:

```yaml
pipeline:
  processors:
    - label: parse_json
      mapping: |
        root = this.parse_json()
    
    - label: debug_parsed
      log:
        level: INFO
        message: '${! content() }'
    
    - label: add_metadata
      mapping: |
        root.processed_at = now()
    
    - label: debug_final
      log:
        level: INFO
        message: '${! content() }'
```

---

## Enable Verbose Logging

These flags control the **edge agent's** own logging. Restart the agent with
verbose logging to see detailed execution information:

```bash
expanso-edge run --verbose
```

**Log levels available:**
- `--log-level trace` - Everything (very detailed)
- `--log-level debug` - Debug information
- `--log-level info` - Standard information (default)
- `--log-level warn` - Warnings only
- `--log-level error` - Errors only

Example with debug level:

```bash
expanso-edge run --log-level debug
```

For one pipeline's log, stream the job instead. It shows records only if the
pipeline has a `log` processor:

```bash
expanso-cli job logs my-job
```

:::note[Console vs File Logging]
When running pipelines, your console only shows **warnings and errors** for pipeline execution logs. All detailed logs (including DEBUG and INFO messages) are written to log files. See [Access Detailed Pipeline Logs](#access-detailed-pipeline-logs) below to learn how to view full debug logs.
:::

---

## Access Detailed Pipeline Logs

When you run pipelines, your terminal stays clean by showing only warnings and errors. But all the detailed debug logs (INFO and DEBUG messages) are still there—they're written to log files on disk so you can dig into them when you need to troubleshoot.

### Where Logs Are Stored

Pipeline logs live in your edge agent's data directory at:

```
{data_dir}/executions/{job_id}/logs/pipeline.log
```

**Default data directory locations:**
- System (root or a system service): `/var/lib/expanso/edge`
- User: `~/.expanso/edge` (on Windows, `%USERPROFILE%\.expanso\edge`)

The agent uses the first writable location.

You can override the data directory using the `--data-dir` flag or `EXPANSO_EDGE_DATA_DIR` environment variable.

### View Pipeline Logs

You have two ways to access logs:

**Option 1: Using the CLI (recommended)**

In the CLI, a pipeline is deployed as a **job**, so you reference it by its pipeline name or id. The CLI gives you formatted log viewing that streams in real time:

```bash
# View pipeline logs (streams in real time)
expanso-cli job logs <job-id>

# Stream from a specific node
expanso-cli job logs <job-id> --node <node-id>
```

This is the quickest way to view logs with nice formatting and real-time tailing.

**Option 2: Direct file access**

If you need to access log files directly (for scripting, archiving, or using other log tools):

```bash
# List executions to find the ID
expanso-cli execution list

# View logs directly
cat ~/.expanso/edge/executions/<job_id>/logs/pipeline.log
```

### Console vs File Logging

Here's how Expanso splits logging between your terminal and disk:

| Output | Log Levels | Use Case |
|--------|-----------|----------|
| **Console** (stdout) | WARN, ERROR only | Quick monitoring, spotting problems |
| **Log files** | All levels (DEBUG, INFO, WARN, ERROR) | Detailed debugging, troubleshooting |

**Why split logs this way?**

Pipeline debug logs can get extremely verbose—showing every message processed, every transformation applied, and so on. By limiting console output to warnings and errors, your terminal stays readable while full details are preserved in files for when you need them.

**Quick guide for which to use:**
- **Console output** - Great when you're running pipelines interactively during development and just want to see that things are working (or catch errors quickly)
- **Log files** - Essential when troubleshooting specific pipeline behavior, investigating data transformation issues, debugging Bloblang mapping logic, or analyzing performance details

---

## Test with Limited Data

When testing, use a fixed number of messages instead of continuous generation.

### Limit Generated Messages

```yaml
input:
  generate:
    count: 10  # Stop after 10 messages
    interval: ""  # Generate as fast as possible
    mapping: |
      root.test_id = uuid_v4()
      root.timestamp = now()
```

### Process Specific Files

Test against a small sample file first. A deployed job reads `paths` on the
edge node that runs it, not on the machine you deploy from, so create the sample
**on that node**, somewhere the agent's user can read:

```bash
# On the edge node: create a test file with 5 records
head -n 5 large-file.json > /tmp/test-data.json
```

Point the pipeline at the sample with an absolute path, then validate and deploy
it:

```yaml
input:
  file:
    paths: [/tmp/test-data.json]
    codec: lines
```

```bash
expanso-edge validate my-pipeline.yaml
expanso-cli job deploy my-job.yaml
```

---

## Common Issues and Solutions

### Issue: File Access Errors

**Symptoms**: `expanso-cli job deploy` or `expanso-cli job validate` fail with errors like "no such file or directory" or "permission denied".

**Cause**: The file path doesn't exist, is inaccessible, or points to a directory instead of a file.

**Solution**: Add the `-v` (verbose) flag to get detailed diagnostics:

```bash
# Default error (minimal details)
expanso-cli job deploy my-job.yaml
# Output: failed to read job specification file: open /path/to/file: no such file or directory

# Verbose mode (detailed diagnostics)
expanso-cli job deploy my-job.yaml -v
```

**Example verbose output:**

```
cannot read job specification '/var/log/app/pipeline.yaml': parent directory does not exist
  path: /var/log/app/pipeline.yaml
  path_exists: no
  parent_exists: no
  parent_path: /var/log/app
```

**What the diagnostic fields mean:**

| Field | Description |
|-------|-------------|
| `path` | The file path you requested |
| `path_exists` | Whether the path exists (yes/no) |
| `parent_exists` | Whether the parent directory exists (yes/no) |
| `parent_path` | The parent directory path |
| `path_type` | Type if path exists (file/directory) |
| `is_symlink` | Whether it's a symbolic link |
| `symlink_target` | Where the symlink points |
| `symlink_broken` | Whether the symlink target is missing |
| `permission_denied` | Whether access was blocked by permissions |

**Common scenarios:**

1. **Parent directory missing** (`parent_exists: no`): Create the directory first, then add your file.

2. **File missing** (`parent_exists: yes`, `path_exists: no`): Check for typos in the filename.

3. **Permission denied** (`permission_denied: yes`): Check file permissions or run as a different user.

4. **Broken symlink** (`symlink_broken: yes`): Fix the symlink target or remove it.

5. **Path is a directory** (`path_type: directory`): You provided a directory path instead of a file path.

---

### Issue: Pipeline Runs But No Output

**Symptoms**: Pipeline starts but nothing appears in output.

**Possible causes:**

1. **Data is being filtered out**

Check your `mapping` processors - are you using `deleted()`?

```yaml
# This will delete all messages!
- mapping: |
    root = if this.status == "active" { deleted() }
```

**Solution**: Add debug output before the filter to see what's being dropped:

```yaml
- label: before_filter
  log:
    level: INFO
    message: '${! content() }'

- mapping: |
    root = if this.status != "active" { deleted() }
```

2. **Input has no data**

**Solution**: Check your input source has data available:

```bash
# For files
ls -lh ./data.json
cat ./data.json

# For generate input - check your mapping is valid
```

3. **Output is going somewhere else**

**Solution**: Temporarily change output to `stdout` for debugging:

```yaml
output:
  stdout:
    codec: lines
```

---

### Issue: Parsing Errors

**Symptoms**: Errors like `failed to parse JSON` or `invalid format`.

**Cause**: Data format doesn't match what you're trying to parse.

**Solution**: Print the raw input first:

```yaml
input:
  file:
    paths: [./data.json]
    codec: lines

pipeline:
  processors:
    # Debug: see raw data
    - label: raw_input
      log:
        level: INFO
        message: '${! content() }'
    
    # Then try parsing
    - mapping: |
        root = this.parse_json()
```

Look at the raw data and adjust your parsing logic accordingly.

---

### Issue: Mapping/Bloblang Errors

**Symptoms**: Errors in transformation logic like `undefined method` or `type mismatch`.

**Cause**: Incorrect Bloblang syntax or attempting operations on wrong data types.

**Solution**: Test transformations incrementally:

```yaml
# Start simple
- mapping: |
    root = this

# Add one transformation at a time
- mapping: |
    root = this
    root.parsed = this.parse_json()

# Validate each step
- log:
    level: INFO
    message: '${! content() }'
```

**Common Bloblang mistakes:**

```
# ❌ Wrong - trying to parse already-parsed JSON
root = this.parse_json().parse_json()

# ✅ Right - parse once
root = this.parse_json()

# ❌ Wrong - field might not exist
root.value = this.data.deeply.nested.field

# ✅ Right - check existence
root.value = this.data.deeply.nested.field | "default"
```

See the [Bloblang Guide](/guides/bloblang) for more patterns.

---

### Issue: Performance - Pipeline is Slow

**Symptoms**: Messages processing slowly, backlog building up.

**Debug approach:**

1. **Add timing measurements**

```yaml
- mapping: |
    root = this
    meta start_time = now()

# ... your processors ...

- mapping: |
    root.processing_time_ms = now().ts_unix_milli() - meta("start_time").ts_unix_milli()
```

2. **Check processor complexity**

Complex transformations in `mapping` processors can slow things down.

**Solution**: Simplify logic or break into multiple simpler processors.

3. **Check output destination**

The output might be the bottleneck (slow API, overloaded database).

**Solution**: Test with `stdout` output to isolate the issue.

---

## Testing Workflows

### 1. Syntax Validation

Always start here:

```bash
expanso-edge validate my-pipeline.yaml
```

### 2. Smoke Test

Run with limited data to stdout:

```yaml
input:
  generate:
    count: 5
    interval: ""
    mapping: |
      root = {"test": "data"}

# ... your processors ...

output:
  stdout:
    codec: lines
```

```bash
expanso-edge validate my-pipeline.yaml
expanso-cli job deploy my-job.yaml
```

### 3. Unit Test Each Component

Test input → processor → output independently:

**Test input only:**

```yaml
input:
  file:
    paths: [./data.json]
    codec: lines

output:
  stdout:
    codec: lines
```

**Test processor only:**

```yaml
input:
  generate:
    count: 5
    interval: ""
    mapping: |
      root = {"test": "data"}

pipeline:
  processors:
    - mapping: |
        root.transformed = this.test.uppercase()

output:
  stdout:
    codec: lines
```

### 4. Integration Test

Test with real data sources but safe outputs (local files or stdout):

```yaml
input:
  http_server:
    address: localhost:8080

pipeline:
  processors:
    - mapping: |
        root = this.parse_json()

output:
  file:
    path: ./test-output.json
    codec: lines
```

### 5. Deploy

Once validated and tested locally, deploy to production via Expanso Cloud.

---

## Debugging Checklist

When something goes wrong, work through this checklist:

- [ ] **Validate syntax**: `expanso-edge validate my-pipeline.yaml`
- [ ] **Check input has data**: Print raw input to stdout
- [ ] **Verify transformations**: Test each processor separately
- [ ] **Enable verbose logging**: Run with `--verbose` or `--log-level debug`
- [ ] **Simplify**: Remove processors one by one to isolate the issue
- [ ] **Test with minimal data**: Use `count: 5` in generate input
- [ ] **Check for filtering**: Ensure you're not accidentally deleting all messages
- [ ] **Validate Bloblang**: Test complex transformations in isolation

---

## Next Steps

- **[Bloblang Guide](/guides/bloblang)** - Learn transformation patterns and functions
- **[`expanso-edge validate` reference](/references/cli/expanso-edge/validate/)** - Full flag and output reference for offline validation
- **[Component Reference](/components)** - Browse all available inputs, processors, and outputs
- **[Error Handling](/guides/pipelines/error_handling)** - Add retry logic and dead letter queues
- **[Deploy to Production](/getting-started/quickstart/)** - Move to managed deployment with Expanso Cloud

:::tip[Learn from Production Examples]
See validation and testing patterns in complete, production-ready pipelines at [examples.expanso.io](https://examples.expanso.io/):
- [Production Pipeline](https://examples.expanso.io/log-processing/production-pipeline/) - Complete production logging setup
- [Circuit Breakers](https://examples.expanso.io/data-routing/circuit-breakers/) - Fault tolerance and resilience patterns
- [Smart Buffering](https://examples.expanso.io/data-routing/smart-buffering/) - Backpressure and buffer management

Each example includes full configuration and testing guidance.
:::

---

## Pro Tips

**Use `generate` for testing**: The `generate` input is your best friend for testing transformations without external dependencies.

**Keep test files small**: Don't test with production-sized data initially. Use 5-10 records.

**Version control your configs**: Track changes to pipeline configurations in git.

**Test transformations in isolation**: Before adding complex Bloblang logic to your pipeline, test it separately:

```yaml
input:
  generate:
    count: 1
    interval: ""
    mapping: |
      root = {"test": "value"}

pipeline:
  processors:
    - mapping: |
        # Test your transformation here
        root.result = this.test.uppercase()

output:
  stdout:
    codec: lines
```

**Use descriptive labels**: Add `label` to all processors to make logs easier to read.

**Start simple, add complexity**: Begin with a minimal pipeline, verify it works, then add features incrementally.

## Troubleshooting

# Troubleshooting

Solutions to common issues when getting started with Expanso.

---

## Node Issues

### Node Not Connecting?

**Symptoms:** Node doesn't appear in Expanso Cloud after bootstrap.

**Solutions:**
- Verify the bootstrap token is correct
- Check network connectivity to cloud.expanso.io
- Review the edge agent logs on the host (`journalctl -u expanso-edge`, or `docker logs` for a container)
- Ensure firewall allows outbound HTTPS connections
- Check system time is synchronized (required for authentication)

**Still not working?**
```bash
# Check whether the agent service is running
systemctl status expanso-edge

# View agent logs
journalctl -u expanso-edge

# Check connectivity with the agent's verbose diagnostics
expanso-edge run --verbose
```

---

## Pipeline Issues

### Pipeline Not Deploying?

**Symptoms:** Pipeline shows "Pending" or fails to deploy.

**Solutions:**
- Ensure the node is online (green status in UI)
- Check for configuration errors in the pipeline YAML
- Review deployment logs in the Expanso Cloud UI
- Verify node has sufficient resources (CPU, memory)

**Debugging steps:**
1. Check node status in the UI
2. View deployment logs for error messages
3. Validate pipeline configuration syntax
4. Check node agent logs for deployment errors

---

### No Data Flowing?

**Symptoms:** Pipeline deployed but no data appears in outputs.

**Solutions:**
- Verify the input source has data available
- Check file paths and permissions for file inputs
- Review processor logic - filters may be dropping messages
- Check edge agent logs for errors

**Debugging steps:**
```bash
# View agent logs
journalctl -u expanso-edge

# Check a specific pipeline's logs
expanso-cli job logs <job-id>

# Verify input source
# For file inputs:
ls -la /path/to/input/files

# For network inputs:
netstat -an | grep PORT
```

---

### Pipeline Readiness Timeout?

**Symptoms:** Pipeline deployment fails with "Pipeline failed to become ready within X" error.

**Solutions:**
- Check the error message—it includes the last specific error that occurred during startup
- Verify input sources are accessible (database running, file paths exist)
- Check for port conflicts if using HTTP server inputs
- Review connection settings for typos (credentials, URLs)
- Check network connectivity to external services
- Review edge agent logs for more details

**Debugging steps:**
```bash
# Check edge agent logs for detailed errors
journalctl -u expanso-edge

# Verify network connectivity to input sources
curl -v http://your-input-source:port

# Check for port conflicts
netstat -an | grep PORT
```

---

## Installation Issues

### Permission Denied During Install

**Solution:** Run install script with appropriate permissions:
```bash
# Linux/macOS
curl -fsSL https://get.expanso.io/edge/install.sh | sudo bash

# Or install to user directory (no sudo)
curl -fsSL https://get.expanso.io/edge/install.sh | PREFIX=$HOME/.local bash
```

### Bootstrap Token Expired

**Symptoms:** "Invalid token" or "Token expired" error.

**Solution:** A bootstrap token's validity is chosen when you create it: 1 hour by default, a longer preset (1 day, 7 days, 30 days, 90 days), or no expiry. If yours expired, generate a new one:
1. Open your workspace's **Keys** page in Expanso Cloud
2. Under **Bootstrap Tokens**, create a new token and copy it
3. Run the bootstrap command again with the new token

---

## Performance Issues

### High CPU Usage

**Causes:**
- High message throughput
- Complex processors (regex, parsing)
- Insufficient batching

**Solutions:**
- Increase batch sizes in processors
- Use simpler processing logic where possible
- Add more nodes to distribute load
- Monitor metrics to identify bottlenecks

### High Memory Usage

**Causes:**
- Large message sizes
- Buffer accumulation
- Memory leaks in custom processors

**Solutions:**
- Monitor the node's memory on the host (for example, `top`, `htop`, or `docker stats`)
- Adjust buffer sizes in pipeline config
- Restart agent if memory grows unbounded
- Check for data accumulation in processors

---

## Network Issues

### Connection Timeouts

**Symptoms:** Agent can't reach cloud.expanso.io.

**Solutions:**
- Verify outbound HTTPS (port 443) is allowed
- Check proxy settings if behind corporate firewall
- Ensure DNS resolution works for cloud.expanso.io
- Run the agent with `--verbose` for network diagnostics (see the Network Diagnostics section below)

### TLS/SSL Errors

**Symptoms:** Certificate verification errors.

**Solutions:**
- Ensure system time is correct
- Update system CA certificates
- Check for corporate SSL inspection/MITM
- Verify no proxy is intercepting HTTPS traffic

---

### Network Diagnostics

Use the `--verbose` flag to get detailed network diagnostics and pinpoint where connectivity is failing.

```bash
# During bootstrap
expanso-edge bootstrap --token YOUR_TOKEN --verbose

# During run
expanso-edge run --verbose
```

**What verbose mode checks:**

- **Resolver**: DNS configuration file in use
- **DNS Servers**: Configured nameservers
- **Proxy**: Whether an HTTP proxy is detected
- **DNS**: Hostname resolution status
- **TCP**: TCP connection to server
- **TLS**: TLS handshake completion

**Example output:**
```
Diagnostics:
- Resolver: /etc/resolv.conf
- DNS Servers: 8.8.8.8, 8.8.4.4
- Proxy: none
- DNS: ok (192.0.2.1)
- TCP: ok (192.0.2.1:443 in 45ms)
- TLS: ok (TLS1.3)
```

**Common diagnostic failures:**

| Diagnostic | Failure | Likely Cause |
|------------|---------|--------------|
| DNS | `failed (no such host)` | DNS resolution issue - check nameservers |
| TCP | `failed (connection refused)` | Firewall blocking outbound connections |
| TCP | `failed (timeout)` | Network unreachable or blocked |
| TLS | `failed (certificate verify failed)` | Corporate proxy intercepting HTTPS |
| Proxy | `using http://proxy:8080` | Traffic routed through proxy - may need `--no-proxy` |

---

## Getting More Help

### Log Collection

When reporting issues, include:
```bash
# Agent version
expanso-edge version

# System info
uname -a

# Recent logs
journalctl -u expanso-edge -n 100

# Node status (as seen by the control plane)
expanso-cli node list
```

### Support Channels

- **Email**: support@expanso.io
- **Documentation**: Check relevant sections:
  - [Quick Start](/getting-started/quickstart/) - Setup walkthrough
  - [Core Concepts](/getting-started/concepts/) - How things work
  - [FAQ](/getting-started/faq) - Common questions

### Reporting Issues

Found a bug or want to request a feature? Use the bug reporter in the Expanso Cloud web UI, or email support@expanso.io.

## Try It Without an Account

# Try It Without an Account

Run a real pipeline on your own machine before you sign up for anything.
Local mode runs the Expanso Edge agent standalone, without a connection to
the control plane, so there is no account, bootstrap token, or Cloud setup.

The pipeline you will run filters log noise at the source. It drops routine
health checks, debug chatter, and successful static-asset requests, and keeps
everything else, including errors and warnings. You send it log lines over
HTTP and see what survives.

## What you need

- `expanso-edge` and `expanso-cli` installed
  ([Installation](/getting-started/installation)). Local mode needs both:
  the agent runs the pipeline, and the CLI submits it.
- `curl` and two terminal windows.

Check both tools are on your `PATH`:

```bash
expanso-edge version
expanso-cli version
```

## Step 1: Write the pipeline

Make an empty folder for this walkthrough and work inside it:

```bash
mkdir expanso-try && cd expanso-try
```

Create `log-filter.yaml`:

```bash
cat > log-filter.yaml <<'EOF'
name: log-filter
description: Keep signal, drop routine log noise at the source
type: pipeline
config:
  input:
    http_server:
      address: 127.0.0.1:8090
      path: /ingest
  pipeline:
    processors:
      - mapping: |
          let raw = content().string()
          let doc = $raw.parse_json().catch(null)
          let doc = if $doc.type() == "object" { $doc } else {
            {"log": $raw}
          }
          let level = $doc.level.string().catch("").lowercase()
          let path = $doc.http.path.string().catch("")
          let status = $doc.http.status.number().catch(0)
          let is_health = $path.re_match(
            "^/(health|healthz|ready|livez|metrics)($|[/?])"
          )
          let is_debug = $level == "debug"
          let is_static = $status > 0 && $status < 400 &&
            $path.re_match("\\.(css|js|png|jpg|svg|woff2?)($|[?#])")
          root = if $is_health || $is_debug || $is_static {
            deleted()
          } else {
            $doc.merge({"kept": true})
          }
  output:
    sync_response: {}
EOF
```

The file is a job: a `name`, a `type`, and the pipeline under `config`.
The pipeline reads each HTTP request body as one log line, decides whether to
keep it with a [`mapping`](/components/processors/mapping) (written in
[Bloblang](/guides/bloblang)), and returns the result in the HTTP response
through [`sync_response`](/components/outputs/sync_response). Lines that are
not JSON objects are kept and wrapped as `{"log": ...}`.

## Step 2: Check it without running anything

```bash
expanso-edge validate log-filter.yaml
```

```
[OK] log-filter.yaml: valid
```

`validate` works offline and never executes the pipeline, so you can run it
after every edit. Add `-o json` to get each error with its path and line
number.

## Step 3: Start a local agent

In your first terminal, from the `expanso-try` folder:

```bash
expanso-edge run --local --data-dir ./.edge
```

Among the startup lines (timestamps trimmed) you will see:

```
INF API server listening addr=127.0.0.1:9010 component=api_server
INF Expanso Edge Service started successfully component=edge_service
```

Leave it running. The agent now accepts jobs on `localhost:9010`.

`--data-dir ./.edge` keeps the agent's job store, execution state, and
pipeline logs in this folder. Without it, the agent uses
`~/.expanso/edge` (or `/var/lib/expanso/edge` when run as a system service).

## Step 4: Deploy the pipeline

In your second terminal, from the same folder:

```bash
export EXPANSO_CLI_CONFIG_DIR="$PWD/.expanso-cli"
expanso-cli job deploy log-filter.yaml \
  --endpoint http://localhost:9010
```

```
Job 'log-filter' created successfully in namespace ''
```

`EXPANSO_CLI_CONFIG_DIR` gives the CLI its own settings folder inside
`expanso-try`, and `--endpoint` names the local agent on every command, so
the CLI never uses a saved profile or an Expanso Cloud workspace you may
have set up elsewhere. Keep both on every `expanso-cli` command in this
walkthrough.

## Step 5: Send it some logs

Create a small sample:

```bash
cat > sample.log <<'EOF'
{"level":"info","http":{"path":"/healthz","status":200}}
{"level":"error","msg":"payment failed"}
{"level":"info","http":{"path":"/orders","status":200}}
{"level":"debug","msg":"cache lookup"}
{"level":"info","http":{"path":"/assets/app.js","status":200}}
{"level":"warn","msg":"retry budget low"}
EOF
```

Send it one line at a time:

```bash
while IFS= read -r line; do
  curl -s -d "$line" http://127.0.0.1:8090/ingest; echo
done < sample.log
```

```

{"kept":true,"level":"error","msg":"payment failed"}
{"http":{"path":"/orders","status":200},"kept":true,"level":"info"}

{"kept":true,"level":"warn","msg":"retry budget low"}
```

Three of the six lines come back. The blank lines are the health check, the
debug line, and the static asset: the pipeline dropped them before they
could go anywhere. The error, the warning, and the real API request
survive, each marked `"kept": true`.

Try a line of your own, in any format:

```bash
curl -s -d 'GET /login 401 from 10.0.0.9' \
  http://127.0.0.1:8090/ingest; echo
```

```
{"kept":true,"log":"GET /login 401 from 10.0.0.9"}
```

## Step 6: Change the rule

Edit `log-filter.yaml`, for example to stop dropping health checks by
removing `$is_health ||` from the `root = if` line. Then check and redeploy
the same file:

```bash
expanso-edge validate log-filter.yaml
expanso-cli job deploy log-filter.yaml \
  --endpoint http://localhost:9010
```

```
[OK] log-filter.yaml: valid
Job 'log-filter' updated successfully in namespace '' (version: 2)
```

Run the Step 5 loop again: the health check now comes back as the
first line.

## Clean up

Stop the agent with `Ctrl+C` in the first terminal. The job store and
pipeline logs are in `./.edge`, so if you start the agent again with the same
`--data-dir`, `log-filter` starts again too. Delete the `expanso-try` folder
(including `.edge` and any `.expanso-cli` settings) to start fresh.

## Troubleshooting

**`bind: address already in use`** when starting the agent means something
else already uses port 9010. Pick another port for the agent:

```bash
expanso-edge run --local --data-dir ./.edge \
  --api-listen 127.0.0.1:9011
```

Then use `--endpoint http://localhost:9011` on every `expanso-cli` command.

The same applies to the pipeline's own port: change `8090` in both
`log-filter.yaml` and the `curl` commands.

**`NO_CHANGES_DETECTED`** from `expanso-cli job deploy` means the file is
identical to the job already deployed. Edit it first, or add `--force` to
deploy it anyway.

**Where is the pipeline's own log?** In
`./.edge/executions/<job-id>/logs/pipeline.log`. Get the job ID with
`expanso-cli job list --endpoint http://localhost:9010`.

## Next steps

- **Examples with a "Try locally" block** that runs on this same agent:
  [data transformer](/examples/data-transformer),
  [metadata enrichment](/examples/metadata-enrichment), and
  [deadband filtering](/examples/deadband-filtering).
- **[Local Mode](/getting-started/local-mode)**: what local mode supports,
  its CLI commands, and troubleshooting.
- **[Bloblang Guide](/guides/bloblang)**: the language behind the `mapping`
  processor.
- **[Cloud Quick Start](/getting-started/quickstart/)**: when you want to
  manage pipelines across a fleet of edge nodes from one control plane.


## Full Reference

See https://docs.expanso.io/getting-started for complete documentation.