Skip to main content

Components

Components are the building blocks of pipelines. Expanso includes 200+ components for connecting to any system.

Component Types​

Inputs​

Sources where data enters the pipeline.

Categories:

  • Messaging: Kafka, NATS, RabbitMQ, MQTT
  • Databases: PostgreSQL, MySQL, MongoDB
  • Files: Local files, S3, SFTP, Azure Blob
  • HTTP: Webhooks, REST APIs
  • Streams: TCP, UDP, WebSocket
  • Special: Generate (testing), Stdin

Example:

input:
  kafka:
    addresses: ["localhost:9092"]
    topics: ["events"]
    consumer_group: "my-app"

Browse all 61 inputs →

Processors​

Transform, filter, and enrich data as it flows through.

Categories:

  • Transformation: Mapping (Bloblang), JQ, JavaScript
  • Parsing: JSON, CSV, XML, Avro, Protobuf, Grok
  • Filtering: Conditional routing, deduplication
  • Enrichment: HTTP requests, cache lookups, SQL queries
  • Aggregation: Batching, windowing, grouping

Example:

pipeline:
  processors:
    - mapping: |
        root.user = this.userId
        root.timestamp = now()
        root.message = this.msg.uppercase()

Browse all 73 processors →

Outputs​

Destinations where processed data is sent.

Categories:

  • Messaging: Kafka, NATS, RabbitMQ, MQTT
  • Databases: PostgreSQL, MySQL, MongoDB, Elasticsearch
  • Storage: S3, GCS, Azure Blob, local files
  • HTTP: REST APIs, webhooks
  • Observability: Datadog, Prometheus, StatsD
  • Special: Stdout (debugging), Drop (discard)

Example:

output:
  s3:
    bucket: processed-data
    path: "year=${!timestamp_year()}/data.json"
    region: us-east-1

Browse all 69 outputs →

Using Components​

Basic Usage​

  1. Choose a component from the catalog
  2. Configure it using YAML
  3. Add to pipeline in the appropriate section

Configuration​

Each component has:

  • Required fields: Must be specified
  • Optional fields: Have sensible defaults
  • Advanced options: For fine-tuning

Finding Components​

By category:

By search:

  • Use the component catalog search
  • Filter by type, status, difficulty

Component Features​

Batching​

Many components support batching for efficiency:

output:
  s3:
    bucket: logs
    batching:
      count: 100
      period: 10s

Error Handling​

Handle failures gracefully:

output:
  kafka:
    addresses: ["localhost:9092"]
    retry:
      max_retries: 3
      backoff:
        initial_interval: 1s

Metrics​

All components export metrics:

  • Throughput (messages/sec, bytes/sec)
  • Latency
  • Errors
  • Success/failure counts

What's Next?​

👉 Browse Components - Explore all 200+ components

👉 Edge Processing - Learn about edge benefits

👉 Examples - See real pipeline examples