Skip to main content

Local Mode

Run Expanso Edge as a standalone API server for rapid pipeline development and testing—no orchestrator or cloud connection required.

What is Local Mode?​

Local mode runs Expanso Edge with a built-in API server that accepts job submissions directly via the CLI. Perfect for:

  • First-time users - Get started without infrastructure
  • Pipeline development - Fast iteration on pipeline configurations
  • Debugging - Test pipelines locally before deployment
  • CI/CD - Automated pipeline testing

What you get:

  • ✅ Full pipeline processing capabilities
  • ✅ REST API for job submission
  • ✅ CLI integration for job management
  • ✅ Local state storage
  • ✅ Zero configuration needed

What you don't get:

  • ❌ Multi-node coordination
  • ❌ Centralized orchestration
  • ❌ Job scheduling and assignment
  • ❌ Fleet management
  • ❌ Web UI for monitoring

Quick Start (5 Minutes)​

Step 1: Start Edge in Local Mode​

expanso-edge run --local

You'll see output like:

INFO Local mode enabled - no orchestrator connection required
INFO API server starting on http://localhost:9010
INFO Ready to accept jobs via API

The edge node is now running and ready to accept pipeline jobs on port 9010.

Step 2: Create a Simple Job​

Create hello-job.yaml:

name: hello-world
description: Simple hello world pipeline
type: pipeline
config:
  input:
    generate:
      count: 5
      interval: "1s"
      mapping: |
        root.message = "Hello from pipeline!"
        root.timestamp = now()
        root.count = count("messages")

  pipeline:
    processors: []

  output:
    stdout:
      codec: lines

Step 3: Submit the Pipeline​

In a new terminal, configure the CLI to point to your local edge, then submit the pipeline:

# Point CLI to local edge
export EXPANSO_CLI_ENDPOINT=http://localhost:9010

# Deploy the job
expanso-cli job deploy hello-job.yaml

You should see:

✓ Job deployed successfully
Job ID: job-abc123

Step 4: Monitor the Job​

Watch the output in the terminal where expanso-edge is running. You'll see:

{"message":"Hello from pipeline!","timestamp":"2024-11-01T12:00:00Z","count":1}
{"message":"Hello from pipeline!","timestamp":"2024-11-01T12:00:01Z","count":2}
{"message":"Hello from pipeline!","timestamp":"2024-11-01T12:00:02Z","count":3}
{"message":"Hello from pipeline!","timestamp":"2024-11-01T12:00:03Z","count":4}
{"message":"Hello from pipeline!","timestamp":"2024-11-01T12:00:04Z","count":5}

🎉 Success! You've run your first pipeline in local mode.

Having issues?

See the Common Errors section below for solutions to job format errors, port conflicts, and connection issues.

CLI Commands for Local Mode​

Configure CLI for Local Edge​

Set the environment variable once to point all CLI commands to your local edge:

export EXPANSO_CLI_ENDPOINT=http://localhost:9010

Tip: Add this to your ~/.bashrc or ~/.zshrc for persistent configuration.

Submit a Job​

expanso-cli job deploy my-pipeline.yaml

Alternative: Use --endpoint flag for one-off commands:

expanso-cli job deploy my-pipeline.yaml --endpoint http://localhost:9010

List Running Jobs​

expanso-cli job list

Get Job Status​

expanso-cli job describe <job-id>

Stop a Job​

expanso-cli job stop <job-id>

View Job Executions​

expanso-cli job executions <job-id>

Configuration​

Custom API Port​

By default, local mode listens on http://localhost:9010. To change:

expanso-edge run --local --api-listen :8080

Then update your environment variable:

export EXPANSO_CLI_ENDPOINT=http://localhost:8080

Data Directory​

Local mode stores state in a local database. Specify a custom location:

expanso-edge run --local --data-dir ./my-data

Enable Debug Logging​

See detailed execution logs:

expanso-edge run --local --log-level debug

Or trace level for even more detail:

expanso-edge run --local --log-level trace

Common Workflows​

Development Iteration Loop​

# Terminal 1: Start edge once
expanso-edge run --local -v

# Terminal 2: Set endpoint and iterate
export EXPANSO_CLI_ENDPOINT=http://localhost:9010

# Deploy v1
expanso-cli job deploy v1-pipeline.yaml
# ... test and observe output ...

# Update pipeline file, deploy v2
expanso-cli job deploy v2-pipeline.yaml
# ... test changes ...

Testing Multiple Pipelines​

Run multiple pipelines concurrently:

# Set endpoint
export EXPANSO_CLI_ENDPOINT=http://localhost:9010

# Submit first pipeline
expanso-cli job deploy pipeline-a.yaml

# Submit second pipeline (runs in parallel)
expanso-cli job deploy pipeline-b.yaml

# List all running jobs
expanso-cli job list

Clean State Between Runs​

Stop edge, remove state, and restart:

# Stop edge (Ctrl+C)

# Remove local state
rm -rf ./data  # or your custom --data-dir path

# Restart fresh
expanso-edge run --local

Examples​

Example 1: File Processing Pipeline​

Process log files locally:

Create logs-job.yaml:

name: log-processor
description: Process log files and extract errors
type: pipeline
config:
  input:
    file:
      paths: ["/var/log/app/*.log"]
      codec: lines

  pipeline:
    processors:
      - mapping: |
          # Parse JSON logs
          root = this.parse_json()

          # Only keep errors
          root = if this.level != "ERROR" { deleted() }

  output:
    file:
      path: ./errors.log
      codec: lines

Run:

# Terminal 1: Start edge
expanso-edge run --local

# Terminal 2: Submit job
export EXPANSO_CLI_ENDPOINT=http://localhost:9010
expanso-cli job deploy logs-job.yaml

Example 2: Data Transformation​

Transform CSV to JSON:

Create csv-transform-job.yaml:

name: csv-to-json
description: Transform CSV users file to JSON
type: pipeline
config:
  input:
    file:
      paths: ["./users.csv"]
      codec: lines

  pipeline:
    processors:
      - mapping: |
          # Skip header row
          root = if @message_number == 0 { deleted() }

          # Parse CSV
          let parts = this.split(",")
          root = {
            "name": $parts.index(0),
            "email": $parts.index(1),
            "age": $parts.index(2).number()
          }

  output:
    file:
      path: ./users.json
      codec: lines

Run:

# Terminal 1: Start edge
expanso-edge run --local

# Terminal 2: Submit job
export EXPANSO_CLI_ENDPOINT=http://localhost:9010
expanso-cli job deploy csv-transform-job.yaml

Example 3: HTTP API Testing​

Test an HTTP input locally:

Create http-receiver-job.yaml:

name: webhook-receiver
description: Receive and process webhook data
type: pipeline
config:
  input:
    http_server:
      address: "0.0.0.0:8081"
      path: /webhook

  pipeline:
    processors:
      - mapping: |
          root = this.parse_json()
          root.processed_at = now()

  output:
    stdout:
      codec: lines

Run:

# Terminal 1: Start edge
expanso-edge run --local

# Terminal 2: Submit job
export EXPANSO_CLI_ENDPOINT=http://localhost:9010
expanso-cli job deploy http-receiver-job.yaml

# Terminal 3: Send test data
curl -X POST http://localhost:8081/webhook \
  -H "Content-Type: application/json" \
  -d '{"event":"test","data":"hello"}'

Common Errors​

Job Format Requirements​

Job Requires Wrapper Structure

When submitting jobs via the CLI or API in local mode, you must use the job wrapper format:

name: my-job            # Required: job name
description: "..."      # Optional: description
type: pipeline          # Required: job type
config:                 # Required: pipeline configuration
  input: { ... }
  pipeline: { ... }     # Required even if empty
  output: { ... }

Valid job types:

  • pipeline - Standard data pipeline (most common)
  • query - One-time query job
  • update - Update existing job
  • config - Configuration job

Common errors:

❌ "job type cannot be empty"

# Missing wrapper - won't work
input:
  generate: { ... }

✅ Solution: Add wrapper structure

name: my-job
type: pipeline
config:
  input:
    generate: { ... }
  pipeline:
    processors: []
  output:
    stdout: {}

❌ "pipeline component is required"

name: my-job
type: pipeline
config:
  input: { ... }
  output: { ... }
  # Missing pipeline section

✅ Solution: Include pipeline section (even if empty)

name: my-job
type: pipeline
config:
  input: { ... }
  pipeline:
    processors: []  # Required even if no processors
  output: { ... }

Note: This wrapper format is specific to job submissions via CLI/API. When using expanso-edge run --config pipeline.yaml (standalone mode covered in Build Your First Pipeline), you use the pipeline config format directly without the wrapper.

Wrong Port Configuration​

Error: Error: failed to connect to edge

Cause: CLI is trying to connect to the wrong port. The default port is 9010, not 8080.

Solution:

# Correct default endpoint
export EXPANSO_CLI_ENDPOINT=http://localhost:9010

# Or if you started edge with custom port:
expanso-edge run --local --api-listen :8080
export EXPANSO_CLI_ENDPOINT=http://localhost:8080

Missing Environment Variable​

Issue: CLI commands fail even though edge is running

Cause: EXPANSO_CLI_ENDPOINT not set, and CLI defaults to cloud endpoint

Solution:

# Set for current session
export EXPANSO_CLI_ENDPOINT=http://localhost:9010

# Or use --endpoint flag for one-off commands
expanso-cli job deploy my-job.yaml --endpoint http://localhost:9010

# Make permanent (add to ~/.bashrc or ~/.zshrc)
echo 'export EXPANSO_CLI_ENDPOINT=http://localhost:9010' >> ~/.bashrc

Troubleshooting​

Port Already in Use​

Error: Error: listen tcp :9010: bind: address already in use

Solution: Use a different port:

# Terminal 1: Start with custom port
expanso-edge run --local --api-listen :8080

# Terminal 2: Update environment variable
export EXPANSO_CLI_ENDPOINT=http://localhost:8080
expanso-cli job deploy my-pipeline.yaml

CLI Can't Connect​

Error: Error: failed to connect to edge

Solutions:

  1. Verify edge is running: Check terminal for "API server starting" message
  2. Check the endpoint: Use --endpoint flag if not using default port
  3. Verify firewall: Ensure localhost connections are allowed

Job Not Running​

Issue: Job submitted but nothing happening

Debug steps:

  1. Check job status:

    export EXPANSO_CLI_ENDPOINT=http://localhost:9010
    expanso-cli job list
    expanso-cli job describe <job-id>

  2. Check edge logs:

    # Restart with debug logging
    expanso-edge run --local --log-level debug

  3. Verify pipeline config:

    # Test with minimal pipeline first

State Issues​

Issue: Old jobs still appearing

Solution: Clean state directory:

# Stop edge
# Remove state
rm -rf ./data  # or your --data-dir path
# Restart
expanso-edge run --local

API Reference​

When running in local mode, Edge exposes these API endpoints:

Submit Job​

POST /api/v1/jobs
Content-Type: application/yaml

[pipeline configuration]

Example:

curl -X POST http://localhost:9010/api/v1/jobs \
  -H "Content-Type: application/yaml" \
  --data-binary @my-pipeline.yaml

List Jobs​

GET /api/v1/jobs

Get Job Details​

GET /api/v1/jobs/{job-id}

Stop Job​

POST /api/v1/jobs/{job-id}/stop

Delete Job​

DELETE /api/v1/jobs/{job-id}

Health Check​

GET /health

Comparison: Local Mode vs. Orchestrated Mode​

FeatureLocal ModeOrchestrated Mode
Setupexpanso-edge run --localRequires orchestrator + bootstrap
Job SubmissionDirect API/CLIVia orchestrator
Multi-nodeNoYes
Job SchedulingManualAutomatic
Web UINoYes (orchestrator)
State StorageLocal BoltDBCentralized
Best ForDevelopment, testingProduction deployments

Next Steps​

Once you're comfortable with local mode:

  1. Deploy to Production - Set up Expanso Cloud and connect edge nodes
  2. Learn Bloblang - Master data transformations
  3. Explore Components - Browse 200+ inputs, processors, and outputs
  4. Testing & Debugging - Advanced troubleshooting techniques

Tips​

Fast Feedback: Local mode starts instantly - no bootstrap, no credentials needed.

Use for CI/CD: Perfect for automated pipeline testing in continuous integration.

Combine Approaches: Use local mode for development, then deploy the same pipeline configurations to production via orchestrator.

Multiple Instances: Run multiple edge instances on different ports for parallel testing:

# Terminal 1
expanso-edge run --local --api-listen :9010

# Terminal 2
expanso-edge run --local --api-listen :9011 --data-dir ./data2

# Terminal 3: Point CLI to first instance
export EXPANSO_CLI_ENDPOINT=http://localhost:9010