Skip to main content

Core Concepts

Learn the fundamental concepts behind Expanso Edge and how it enables distributed data pipeline orchestration.

What is Expanso?​

Expanso Edge is a managed platform for deploying and orchestrating data pipelines at the edge. Process data where it's generated, reducing bandwidth costs and latency while maintaining centralized control.

Key capabilities:

  • Edge-native processing at the data source
  • Managed SaaS with no infrastructure to maintain
  • Visual pipeline builder for rapid development
  • 200+ built-in components for any use case

Architecture Overview​

Core Components​

1. Expanso Cloud (Control Plane)​

The Control Plane is the central management interface hosted at cloud.expanso.io.

Responsibilities:

  • Pipeline creation and configuration
  • Agent deployment and monitoring
  • Metrics aggregation and visualization
  • User authentication and authorization
  • Network management

You don't run this - it's a managed service provided by Expanso.

2. Networks​

A Network is a logical grouping of agents that work together.

Key Features:

  • Isolates agents and pipelines
  • Simplifies management and organization
  • Enables team collaboration
  • Supports label-based deployment

Example Use Cases:

  • One network per environment (dev, staging, production)
  • One network per region (us-east, eu-west, ap-south)
  • One network per team or project

Creating a Network:

# Via CLI
expanso network create my-network

# Or use Expanso Cloud UI

3. Agents​

An Agent is the runtime that executes pipelines on your infrastructure.

Characteristics:

  • Lightweight Go binary (~50MB)
  • Runs on Linux, macOS, Windows
  • Connects to Expanso Cloud over TLS
  • Executes one or more pipelines
  • Reports metrics and health

Agent Lifecycle:

  1. Bootstrap: Register with Expanso Cloud using a token
  2. Connect: Establish secure connection to control plane
  3. Receive: Download pipeline configurations
  4. Execute: Run pipelines and process data
  5. Report: Send metrics and logs to control plane

Deployment Options:

  • Bare metal servers
  • Virtual machines
  • Docker containers
  • Kubernetes pods
  • Edge devices (Raspberry Pi, IoT gateways)

4. Pipelines​

A Pipeline defines how data flows from inputs, through processors, to outputs.

Pipeline Structure:

input:
  # Where data comes from
  kafka:
    addresses: ["localhost:9092"]
    topics: ["logs"]

pipeline:
  processors:
    # What to do with the data
    - mapping: |
        root.message = this.msg.uppercase()
        root.timestamp = now()

output:
  # Where data goes
  s3:
    bucket: processed-logs
    path: "logs/${!timestamp_unix()}.json"

Pipeline Components:

Inputs​

Receive data from external sources:

  • Message queues (Kafka, RabbitMQ, NATS)
  • Databases (PostgreSQL, MongoDB)
  • Files (local, S3, SFTP)
  • HTTP servers (webhooks, APIs)
  • Streams (TCP, UDP, WebSocket)

Processors​

Transform, filter, and enrich data:

  • Mapping: Transform with Bloblang language
  • Filtering: Drop unwanted messages
  • Aggregation: Batch, window, group
  • Enrichment: Lookup external data
  • Parsing: JSON, CSV, XML, Avro, Protobuf

Outputs​

Send processed data to destinations:

  • Message queues (Kafka, RabbitMQ, NATS)
  • Databases (PostgreSQL, Elasticsearch)
  • Object storage (S3, GCS, Azure Blob)
  • HTTP endpoints (webhooks, APIs)
  • Observability platforms (Datadog, Prometheus)

Data Flow​

Simple Pipeline Flow​

Multi-Output Pipeline​

Key Concepts in Depth​

Edge-First Processing​

Traditional approach:

Edge Device → Cloud → Process → Store
             ↑
         High cost, High latency

Expanso approach:

Edge Device → Process at Edge → Cloud (filtered/aggregated)
                    ↑
            Low cost, Low latency

Benefits:

  • Reduced bandwidth by filtering at source
  • Lower latency through local processing
  • Less data transfer and cloud processing costs
  • Keep sensitive data on-premises
  • Continue processing during network outages

Bloblang Mapping Language​

Bloblang is Expanso's transformation language for manipulating data.

Example:

# Parse and transform log entry
root.level = this.severity.uppercase()
root.message = this.msg
root.timestamp = this.time.parse_timestamp("2006-01-02")
root.user_id = this.user.id.string()

# Add metadata
root.processed_at = now()
root.source = "expanso-edge"

# Conditional logic
root.alert = if this.severity == "ERROR" {
  true
} else {
  false
}

See the Bloblang Guide for more details.

Labels and Selectors​

Use labels to organize and target agents for deployment.

Label Examples:

env: production
region: us-east-1
datacenter: dc-1
role: log-processor

Deployment with Selectors:

# Deploy only to production agents in us-east-1
selector:
  env: production
  region: us-east-1

This enables:

  • Gradual rollouts
  • Environment-specific deployments
  • Geographic targeting
  • Role-based deployments

Job Distribution and Scheduling​

When you deploy a job, Expanso's scheduler figures out which nodes should run it and creates executions on those nodes. This is how your pipelines spread across your infrastructure.

How Jobs Run on Nodes​

Each node runs at most one execution of any given job. When you deploy a job with a selector, the scheduler:

  1. Finds all nodes that match the selector labels
  2. Creates exactly one execution on each matching node
  3. Monitors the executions and keeps them running

If you have 10 nodes matching your selector, you get 10 executions of the same pipeline—one per node.

name: log-processor
type: pipeline
selector:
  match_labels:
    env: production
    region: us-west
config:
  # ... your pipeline config ...

With 5 nodes labeled env: production and region: us-west, this job creates 5 executions—one on each node.

Automatic Scaling​

Jobs automatically scale up and down as your fleet changes. No manual intervention required.

When new nodes join that match your job's selector, the scheduler detects them and starts executions automatically. Your job scales from N executions to N+1. When nodes disconnect, their executions stop and the job continues running on remaining nodes. When disconnected nodes come back online, their executions resume.

Say you start with 3 nodes running a log processing job. You add 2 more nodes with matching labels. The scheduler automatically deploys the job to the new nodes, scaling from 3 to 5 executions with no action on your part.

Selective Placement with Labels​

Label selectors give you fine-grained control over where jobs run.

To deploy to all nodes in a region:

selector:
  match_labels:
    region: us-east-1

This deploys to every node with region: us-east-1, regardless of other labels.

To match nodes with multiple criteria:

selector:
  match_labels:
    env: production
    tier: premium
    role: analytics

This deploys only to nodes that have ALL three labels. Nodes missing any label won't receive executions.

To deploy to all nodes in the network:

# No selector field means "all nodes"
name: fleet-monitor
type: pipeline
config:
  # ... pipeline config ...

Without a selector, the job deploys to every node in the network.

Practical Implications​

For capacity planning:

  • If you have 100 edge nodes and deploy a job without selectors, you get 100 executions
  • Each execution consumes resources (CPU, memory) on its node
  • Plan resource requirements per-node, not per-job

For gradual rollouts:

  • Start with a narrow selector (e.g., env: dev)
  • Test with a small subset of nodes
  • Expand the selector to deploy to more nodes
  • The scheduler handles the scaling automatically

For geographic distribution:

  • Use region labels to control where data is processed
  • Deploy region-specific jobs with region selectors
  • Jobs automatically scale within their region as nodes join

For multi-tenant scenarios:

  • Use customer/tenant labels on nodes
  • Deploy customer-specific jobs with customer selectors
  • Each customer's job runs only on their nodes
  • Automatic isolation through label-based placement

State and Persistence​

Agents maintain state for:

  • Checkpoints: Resume from last processed position
  • Caches: Store lookup data locally
  • Deduplication: Track processed messages
  • Aggregations: Maintain windows and counters

Storage Options:

  • Local disk (default)
  • Memory (fast, not persistent)
  • External (Redis, etcd)

Deployment Patterns​

Pattern 1: Edge Filtering​

Filter data at the edge, send only relevant data to cloud.

Edge Agent: Filter 95% of noise → Cloud: Store 5% of important data

Use Case: IoT sensors generating high-frequency data

Pattern 2: Regional Aggregation​

Aggregate data regionally, send summaries to central location.

Region 1 Agent: Aggregate hourly → Central Storage
Region 2 Agent: Aggregate hourly → Central Storage

Use Case: Multi-location retail analytics

Pattern 3: Hub and Spoke​

Edge agents send to regional hubs, hubs aggregate and forward.

Edge Agents → Regional Hub → Central Cloud

Use Case: Manufacturing facilities with central monitoring

Pattern 4: Mesh Processing​

Agents communicate directly for distributed processing.

Agent 1 ↔ Agent 2 ↔ Agent 3 → Distributed Result

Use Case: Distributed machine learning inference

Security Model​

Authentication​

  • Agents authenticate using bootstrap tokens
  • Tokens are single-use and time-limited
  • mTLS for all agent-cloud communication

Authorization​

  • Role-based access control (RBAC)
  • Network-level isolation
  • Pipeline deployment permissions

Data Security​

  • End-to-end encryption in transit
  • At-rest encryption for agent state
  • PII redaction at the edge
  • Audit logging

Monitoring and Observability​

Metrics​

Agents export metrics:

  • Throughput: Messages/sec, bytes/sec
  • Latency: Processing time, queue depth
  • Errors: Error rates, failed messages
  • Resources: CPU, memory, disk usage

Logs​

Three levels of logs:

  • Pipeline logs: Data flowing through pipelines
  • Agent logs: Agent operations and health
  • Audit logs: User actions and deployments

Health Checks​

Agents report health every 10 seconds:

  • Connection status
  • Pipeline status
  • Resource availability
  • Error conditions

What's Next?​

Now that you understand the core concepts: