Testing & Debugging

Learn how to validate configurations, add debugging output, and troubleshoot common issues when building pipelines.

Quick Reference

# Validate a pipeline config offline (no control plane needed)
expanso-edge validate my-pipeline.yaml

# Validate a job spec against the control plane
expanso-cli job validate my-pipeline.yaml

# Run pipeline locally with verbose logging
expanso-edge run --config my-pipeline.yaml --verbose

# Test with limited data
# (add count: 10 to your generate input)

# Output to terminal for inspection
# (use stdout output in your config)

---

Editor Setup

Get autocomplete and validation while writing pipeline configs - makes authoring faster and catches errors before you run anything.

Expanso provides a JSON Schema for pipeline YAML files. Point your editor at it and you'll get autocomplete for component names, validation for config fields, and inline docs without leaving your editor.

What you'll get:

• Browse available components as you type (inputs, processors, outputs)
• Catch typos and missing fields immediately
• See component docs on hover
• Write configs faster with fewer errors

Configure VS Code

Add this to .vscode/settings.json in your workspace (or global settings):

{
  "yaml.schemas": {
    "https://docs.expanso.io/schemas/pipeline.schema.json": [
      "**/*.pipeline.yaml",
      "**/*pipeline*.yaml"
    ]
  }
}

This maps the Expanso pipeline schema to any YAML files matching these patterns.

Per-workspace setup (recommended):

1. Create .vscode/settings.json in your project directory
2. Add the schema configuration above
3. Commit it to version control so your team gets autocomplete automatically

Global setup (applies to all projects):

1. Open VS Code settings (Cmd+, or Ctrl+,)
2. Search for "yaml.schemas"
3. Click "Edit in settings.json"
4. Add the schema configuration

Configure IntelliJ IDEA / PyCharm

1. Go to Preferences → Languages & Frameworks → Schemas and DTDs → JSON Schema Mappings
2. Click + to add a new schema
3. Set:
   • Name: Expanso Pipeline Schema
   • Schema file or URL: https://docs.expanso.io/schemas/pipeline.schema.json
   • Schema version: JSON Schema version 7
4. Add file path patterns:
   • **/*.pipeline.yaml
   • **/*pipeline*.yaml
5. Click OK

Configure Other Editors

Most editors with YAML support can use JSON Schema for validation.

Neovim (with yaml-language-server):

Add to your LSP config or .luarc.json:

{
  "yaml.schemas": {
    "https://docs.expanso.io/schemas/pipeline.schema.json": "**/*.pipeline.yaml"
  }
}

Sublime Text (with LSP-yaml):

Add to LSP-yaml settings:

{
  "settings": {
    "yaml.schemas": {
      "https://docs.expanso.io/schemas/pipeline.schema.json": "**/*.pipeline.yaml"
    }
  }
}

General approach: Look for YAML language server or JSON Schema support in your editor's docs. Configure it to associate https://docs.expanso.io/schemas/pipeline.schema.json with your pipeline YAML files.

What You'll See

Once configured:

Autocomplete for components:

Start typing under input:, processors:, or output: and you'll get suggestions for all available components (kafka, http_server, generate, mapping, etc.).

Parameter completion:

When configuring a component, autocomplete shows valid configuration fields. For example, typing under kafka: suggests addresses, topics, consumer_group, and other Kafka-specific options.

Inline validation:

Red squiggly lines appear immediately when you:

• Misspell a component name
• Forget required fields
• Use invalid configuration keys
• Have incorrect YAML indentation

Hover documentation:

Hover over any component or field to see its description and usage notes without switching to the docs.

File Naming

The schema works best with these naming patterns:

• *.pipeline.yaml - Standard pipeline configs
• log-processor.pipeline.yaml - Descriptive names
• my-pipeline.yaml - Also works if you add the pattern to your schema config

You can customize the file patterns in your editor's schema configuration to match your team's naming conventions.

Limitations

The JSON Schema provides syntax validation and autocomplete, but it doesn't validate:

• Bloblang expressions: Your mapping logic syntax isn't checked
• Runtime values: Environment variables like ${VAR} aren't validated
• Component availability: All components show in autocomplete, but some may not be available in your Expanso Edge version

Always validate before you run the pipeline:

expanso-edge validate my-pipeline.yaml

This catches issues the schema can't detect, like invalid Bloblang syntax or a field the component doesn't accept.

Tips

Use descriptive file names: log-processor.pipeline.yaml is clearer than pipeline1.yaml.

Commit your workspace settings: Share .vscode/settings.json with your team so everyone gets autocomplete automatically.

Validate often: Run expanso-edge validate to catch issues the schema misses. It needs no server, so it's cheap to run on every save.

---

Validate Before Running

Always validate your pipeline configuration before running it. This catches syntax errors and configuration mistakes early.

Two commands do this, and they check different things:

Command	Needs a server	What it checks
expanso-edge validate	No	Pipeline config in depth: structure, component fields, Bloblang mappings
expanso-cli job validate --offline	No	Job spec syntax and structure
expanso-cli job validate	Yes	Job spec plus comprehensive checks against the control plane

Start with expanso-edge validate while writing the config, then validate the job spec before you deploy.

Offline Validation on the Node

expanso-edge validate is the deepest check you can run without a server. It runs entirely on the machine you invoke it on — no control plane connection, no node credentials, and no edge configuration required — which makes it useful both as a pre-flight check on an edge node and as a CI step.

expanso-edge validate my-pipeline.yaml

[OK] my-pipeline.yaml: valid

What it checks:

• ✅ Valid YAML (or JSON)
• ✅ Config structure - unknown fields, wrong types, unsupported top-level sections
• ✅ Component names exist and accept the fields you set
• ✅ Bloblang mapping syntax

What it does not check:

• ❌ Job-level fields - name, type, namespace, selectors, rollout, restart policy
• ❌ Control plane admission - namespace existence, node matching, server dry-run

A config that passes expanso-edge validate can still be rejected when you deploy it, so validate the job spec as well before submitting.

Reading the Output

Failures list one line per problem, with a position in the file where one is known:

[FAIL] my-pipeline.yaml: 2 error(s)
  - (line 7) bloblang syntax error in input.generate.mapping. Expected method or field path (unexpected end of expression)
  - (line 2, col 3) [input.generate]. Unknown field 'bogus_field' in generate component. Check the documentation for valid fields in the 'generate' component.
  hint: Check your Bloblang mapping syntax and pipeline configuration structure.

Each error names the path within the config (input.generate.mapping), so you can jump straight to the offending key.

Input Sources

Validate files, piped input, or several configs at once:

# One or more files
expanso-edge validate pipeline.yaml

# Several files - each is reported separately
expanso-edge validate pipelines/*.yaml

# Piped input (reads stdin when given "-" or no file at all)
cat pipeline.yaml | expanso-edge validate -

A single file or stream can hold multiple YAML documents separated by ---. Each document is validated independently and labeled with its position:

[OK] pipelines.yaml (doc 1): valid
[FAIL] pipelines.yaml (doc 2): 1 error(s)
  - (line 4) bloblang syntax error in input.generate.mapping. Expected query (unexpected end of expression)
  hint: Check your Bloblang mapping syntax. See documentation for valid expressions.

Job Specs

You can pass a full job spec instead of a bare config, as a convenient way to check the config it carries. The spec itself is only checked for parsing - it must be valid YAML or JSON that reads as a job - and then the embedded config block gets the same full validation a bare config does, with reported positions pointing into that embedded config. Job-level fields are not checked; use expanso-cli job validate for those.

expanso-edge validate hello-world-job.yaml

Two behaviors are worth knowing:

• A pipeline job with a missing or misspelled config key is an error, not a pass:

  [FAIL] hello-world-job.yaml: 1 error(s)
    - pipeline job has no config to validate (is the 'config' key present and correctly spelled?)

• Job types that carry no pipeline config pass with a note:

  [OK] node-check.yaml: valid (query job; no pipeline config to validate)

Exit Codes and Scripting

The command exits 0 only when every document in every input is valid, and non-zero if anything is invalid, empty, or unreadable - so it composes with the rest of your shell without extra parsing:

expanso-edge validate pipelines/*.yaml

Add --output json (or -o json) for machine-readable results. Validation results go to stdout and the failure summary goes to stderr, so you can pipe straight into jq:

expanso-edge validate pipelines/*.yaml --output json | jq -r '.[] | select(.valid == false) | .source'

[
  {
    "source": "bad.yaml",
    "valid": false,
    "kind": "config",
    "hint": "Check your Bloblang mapping syntax and pipeline configuration structure.",
    "errors": [
      {
        "kind": "bloblang",
        "path": "input.generate.mapping",
        "message": "Expected method or field path (unexpected end of expression)",
        "line": 7
      },
      {
        "kind": "schema",
        "path": "input.generate",
        "message": "Unknown field 'bogus_field' in generate component",
        "suggestion": "Check the documentation for valid fields in the 'generate' component.",
        "line": 2,
        "column": 3,
        "component": "generate"
      }
    ]
  }
]

Each result carries source, valid, and kind (config, pipeline-job, job, or error), plus note and, when invalid, hint and errors. Each error carries kind (schema, bloblang, policy, or yaml), message, and - where known - path, suggestion, line, column, component, and label.

Mappings that read the local machine

Validation happens where the pipeline would run, so Bloblang functions that read host state are resolved against the machine running the command. A literal file("/etc/expanso/token") in a mapping is read at validation time and reports an error if that path is missing locally - which is what you want on the target node, but means a config can fail on a CI runner and pass on the node (or the reverse). Environment interpolation like ${VAR} in config fields is not resolved during validation.

Validating Job Specs with the CLI

expanso-cli job validate checks the job spec itself - the fields expanso-edge validate deliberately skips.

Client-side only, no server needed:

expanso-cli job validate my-pipeline.yaml --offline

Example output (success, offline):

✓ Client-side validation passed
Note: Server-side validation skipped (--offline mode)

Example output (error):

Error: pipeline 'input' component is required

For comprehensive validation against your Expanso Cloud environment, drop --offline:

expanso-cli job validate my-pipeline.yaml

This performs both client-side and server-side checks (requires connection to Expanso Cloud).

---

Add Debug Output

When developing pipelines, add log processors at each stage to see what's happening.

Debug at Each Stage

input:
  file:
    paths: [./data.json]
    codec: lines

pipeline:
  processors:
    # First transformation
    - mapping: |
        root = this.parse_json()
    
    # Debug: log what we parsed
    - log:
        level: INFO
        message: '${! content() }'
    
    # Second transformation
    - mapping: |
        root.processed = true
        root.timestamp = now()

output:
  stdout:
    codec: lines

Use Labels for Clarity

Add labels to track which stage produced output:

pipeline:
  processors:
    - label: parse_json
      mapping: |
        root = this.parse_json()
    
    - label: debug_parsed
      log:
        level: INFO
        message: '${! content() }'
    
    - label: add_metadata
      mapping: |
        root.processed_at = now()
    
    - label: debug_final
      log:
        level: INFO
        message: '${! content() }'

---

Enable Verbose Logging

Run with verbose logging to see detailed execution information:

expanso-edge run --config my-pipeline.yaml --verbose

Log levels available:

• --log-level trace - Everything (very detailed)
• --log-level debug - Debug information
• --log-level info - Standard information (default)
• --log-level warn - Warnings only
• --log-level error - Errors only

Example with debug level:

expanso-edge run --config my-pipeline.yaml --log-level debug

Console vs File Logging

When running pipelines, your console only shows warnings and errors for pipeline execution logs. All detailed logs (including DEBUG and INFO messages) are written to log files. See Access Detailed Pipeline Logs below to learn how to view full debug logs.

---

Access Detailed Pipeline Logs

When you run pipelines, your terminal stays clean by showing only warnings and errors. But all the detailed debug logs (INFO and DEBUG messages) are still there—they're written to log files on disk so you can dig into them when you need to troubleshoot.

Where Logs Are Stored

Pipeline logs live in your edge agent's data directory at:

{data_dir}/executions/{job_id}/logs/pipeline.log

Default data directory locations:

• System (root or a system service): /var/lib/expanso/edge
• User: ~/.expanso/edge (on Windows, %USERPROFILE%\.expanso\edge)

The agent uses the first writable location.

You can override the data directory using the --data-dir flag or EXPANSO_EDGE_DATA_DIR environment variable.

View Pipeline Logs

You have two ways to access logs:

Option 1: Using the CLI (recommended)

In the CLI, a pipeline is deployed as a job, so you reference it by its pipeline name or id. The CLI gives you formatted log viewing that streams in real time:

# View pipeline logs (streams in real time)
expanso-cli job logs <job-id>

# Stream from a specific node
expanso-cli job logs <job-id> --node <node-id>

This is the quickest way to view logs with nice formatting and real-time tailing.

Option 2: Direct file access

If you need to access log files directly (for scripting, archiving, or using other log tools):

# List executions to find the ID
expanso-cli execution list

# View logs directly
cat ~/.expanso/edge/executions/<job_id>/logs/pipeline.log

Console vs File Logging

Here's how Expanso splits logging between your terminal and disk:

Output	Log Levels	Use Case
Console (stdout)	WARN, ERROR only	Quick monitoring, spotting problems
Log files	All levels (DEBUG, INFO, WARN, ERROR)	Detailed debugging, troubleshooting

Why split logs this way?

Pipeline debug logs can get extremely verbose—showing every message processed, every transformation applied, and so on. By limiting console output to warnings and errors, your terminal stays readable while full details are preserved in files for when you need them.

Quick guide for which to use:

• Console output - Great when you're running pipelines interactively during development and just want to see that things are working (or catch errors quickly)
• Log files - Essential when troubleshooting specific pipeline behavior, investigating data transformation issues, debugging Bloblang mapping logic, or analyzing performance details

---

Test with Limited Data

When testing, use a fixed number of messages instead of continuous generation.

Limit Generated Messages

input:
  generate:
    count: 10  # Stop after 10 messages
    interval: ""  # Generate as fast as possible
    mapping: |
      root.test_id = uuid_v4()
      root.timestamp = now()

Process Specific Files

Test against a small sample file first:

# Create test file with 5 records
head -n 5 large-file.json > test-data.json

# Test against small file first
expanso-edge run --config my-pipeline.yaml

input:
  file:
    paths: [./test-data.json]
    codec: lines

---

Common Issues and Solutions

Issue: File Access Errors

Symptoms: expanso-cli job deploy or expanso-cli job validate fail with errors like "no such file or directory" or "permission denied".

Cause: The file path doesn't exist, is inaccessible, or points to a directory instead of a file.

Solution: Add the -v (verbose) flag to get detailed diagnostics:

# Default error (minimal details)
expanso-cli job deploy my-pipeline.yaml
# Output: failed to read job specification file: open /path/to/file: no such file or directory

# Verbose mode (detailed diagnostics)
expanso-cli job deploy my-pipeline.yaml -v

Example verbose output:

cannot read job specification '/var/log/app/pipeline.yaml': parent directory does not exist
  path: /var/log/app/pipeline.yaml
  path_exists: no
  parent_exists: no
  parent_path: /var/log/app

What the diagnostic fields mean:

Field	Description
path	The file path you requested
path_exists	Whether the path exists (yes/no)
parent_exists	Whether the parent directory exists (yes/no)
parent_path	The parent directory path
path_type	Type if path exists (file/directory)
is_symlink	Whether it's a symbolic link
symlink_target	Where the symlink points
symlink_broken	Whether the symlink target is missing
permission_denied	Whether access was blocked by permissions

Common scenarios:

1. Parent directory missing (parent_exists: no): Create the directory first, then add your file.

2. File missing (parent_exists: yes, path_exists: no): Check for typos in the filename.

3. Permission denied (permission_denied: yes): Check file permissions or run as a different user.

4. Broken symlink (symlink_broken: yes): Fix the symlink target or remove it.

5. Path is a directory (path_type: directory): You provided a directory path instead of a file path.

---

Issue: Pipeline Runs But No Output

Symptoms: Pipeline starts but nothing appears in output.

Possible causes:

1. Data is being filtered out

Check your mapping processors - are you using deleted()?

# This will delete all messages!
- mapping: |
    root = if this.status == "active" { deleted() }

Solution: Add debug output before the filter to see what's being dropped:

- label: before_filter
  log:
    level: INFO
    message: '${! content() }'

- mapping: |
    root = if this.status != "active" { deleted() }

2. Input has no data

Solution: Check your input source has data available:

# For files
ls -lh ./data.json
cat ./data.json

# For generate input - check your mapping is valid

3. Output is going somewhere else

Solution: Temporarily change output to stdout for debugging:

output:
  stdout:
    codec: lines

---

Issue: Parsing Errors

Symptoms: Errors like failed to parse JSON or invalid format.

Cause: Data format doesn't match what you're trying to parse.

Solution: Print the raw input first:

input:
  file:
    paths: [./data.json]
    codec: lines

pipeline:
  processors:
    # Debug: see raw data
    - label: raw_input
      log:
        level: INFO
        message: '${! content() }'
    
    # Then try parsing
    - mapping: |
        root = this.parse_json()

Look at the raw data and adjust your parsing logic accordingly.

---

Issue: Mapping/Bloblang Errors

Symptoms: Errors in transformation logic like undefined method or type mismatch.

Cause: Incorrect Bloblang syntax or attempting operations on wrong data types.

Solution: Test transformations incrementally:

# Start simple
- mapping: |
    root = this

# Add one transformation at a time
- mapping: |
    root = this
    root.parsed = this.parse_json()

# Validate each step
- log:
    level: INFO
    message: '${! content() }'

Common Bloblang mistakes:

# ❌ Wrong - trying to parse already-parsed JSON
root = this.parse_json().parse_json()

# ✅ Right - parse once
root = this.parse_json()

# ❌ Wrong - field might not exist
root.value = this.data.deeply.nested.field

# ✅ Right - check existence
root.value = this.data.deeply.nested.field | "default"

See the Bloblang Guide for more patterns.

---

Issue: Performance - Pipeline is Slow

Symptoms: Messages processing slowly, backlog building up.

Debug approach:

1. Add timing measurements

- mapping: |
    root = this
    meta start_time = now()

# ... your processors ...

- mapping: |
    root.processing_time_ms = now().ts_unix_milli() - meta("start_time").ts_unix_milli()

2. Check processor complexity

Complex transformations in mapping processors can slow things down.

Solution: Simplify logic or break into multiple simpler processors.

3. Check output destination

The output might be the bottleneck (slow API, overloaded database).

Solution: Test with stdout output to isolate the issue.

---

Testing Workflows

1. Syntax Validation

Always start here:

expanso-edge validate my-pipeline.yaml

2. Smoke Test

Run with limited data to stdout:

input:
  generate:
    count: 5
    interval: ""
    mapping: |
      root = {"test": "data"}

# ... your processors ...

output:
  stdout:
    codec: lines

expanso-edge run --config my-pipeline.yaml

3. Unit Test Each Component

Test input → processor → output independently:

Test input only:

input:
  file:
    paths: [./data.json]
    codec: lines

output:
  stdout:
    codec: lines

Test processor only:

input:
  generate:
    count: 5
    interval: ""
    mapping: |
      root = {"test": "data"}

pipeline:
  processors:
    - mapping: |
        root.transformed = this.test.uppercase()

output:
  stdout:
    codec: lines

4. Integration Test

Test with real data sources but safe outputs (local files or stdout):

input:
  http_server:
    address: localhost:8080

pipeline:
  processors:
    - mapping: |
        root = this.parse_json()

output:
  file:
    path: ./test-output.json
    codec: lines

5. Deploy

Once validated and tested locally, deploy to production via Expanso Cloud.

---

Debugging Checklist

When something goes wrong, work through this checklist:

• Validate syntax: expanso-edge validate my-pipeline.yaml
• Check input has data: Print raw input to stdout
• Verify transformations: Test each processor separately
• Enable verbose logging: Run with --verbose or --log-level debug
• Simplify: Remove processors one by one to isolate the issue
• Test with minimal data: Use count: 5 in generate input
• Check for filtering: Ensure you're not accidentally deleting all messages
• Validate Bloblang: Test complex transformations in isolation

---

Next Steps

• Bloblang Guide - Learn transformation patterns and functions
• expanso-edge validate reference - Full flag and output reference for offline validation
• Component Reference - Browse all available inputs, processors, and outputs
• Error Handling - Add retry logic and dead letter queues
• Deploy to Production - Move to managed deployment with Expanso Cloud

Learn from Production Examples

See validation and testing patterns in complete, production-ready pipelines at examples.expanso.io:

• Production Pipeline - Complete production logging setup
• Circuit Breakers - Fault tolerance and resilience patterns
• Smart Buffering - Backpressure and buffer management

Each example includes full configuration and testing guidance.

---

Pro Tips

Use generate for testing: The generate input is your best friend for testing transformations without external dependencies.

Keep test files small: Don't test with production-sized data initially. Use 5-10 records.

Version control your configs: Track changes to pipeline configurations in git.

Test transformations in isolation: Before adding complex Bloblang logic to your pipeline, test it separately:

input:
  generate:
    count: 1
    interval: ""
    mapping: |
      root = {"test": "value"}

pipeline:
  processors:
    - mapping: |
        # Test your transformation here
        root.result = this.test.uppercase()

output:
  stdout:
    codec: lines

Use descriptive labels: Add label to all processors to make logs easier to read.

Start simple, add complexity: Begin with a minimal pipeline, verify it works, then add features incrementally.