Inputs
An input is a source of data piped through an array of optional processors:
input:
label: my_redis_input
redis_streams:
url: tcp://localhost:6379
streams:
- expanso_stream
body_key: body
consumer_group: expanso_group
# Optional list of processing steps
processors:
- mapping: |
root.document = this.without("links")
root.link_count = this.links.length()
Some inputs have a logical end, for example a csv input ends once the last row is consumed, when this happens the input gracefully terminates and Expanso Edge will shut itself down once all messages have been processed fully.
Brokering
Only one input is configured at the root of a pipeline config. However, the root input can be a broker which combines multiple inputs and merges the streams:
input:
broker:
inputs:
- kafka:
addresses: [ TODO ]
topics: [ foo, bar ]
consumer_group: foogroup
- redis_streams:
url: tcp://localhost:6379
streams:
- expanso_stream
body_key: body
consumer_group: expanso_group
Labels
Inputs have an optional field label that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple inputs, otherwise their metrics labels will be generated based on their composition.
Database Inputs
Query databases to read data into your pipeline:
- sql_select: Query database tables (MySQL, PostgreSQL, SQLite, etc.)
- sql_raw: Execute custom SQL queries
- mongodb: Query MongoDB collections
- cassandra: Query Cassandra tables
See the Database Connectivity Guide for complete examples including:
- Reading from PostgreSQL and MySQL
- Using SQLite for edge caching and analytics
- Database-to-database replication
- Cloud database authentication (AWS RDS, Azure)
Browse Inputs
Showing 60 of 200 components
amqp_0_9
Input
Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ.
amqp_1
Input
Reads messages from an AMQP (1.0) server.
aws_kinesis
Input
Receive messages from one or more Kinesis streams.
aws_s3
Input
Downloads objects within an Amazon S3 bucket, optionally filtered by a prefix, either by walking the items in the bucket or by streaming upload notifi...
aws_sqs
Input
Consume messages from an AWS SQS URL.
azure_blob_storage
Input
Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix.
azure_cosmosdb
Input
Executes a SQL query against Azure CosmosDB and creates a batch of messages from each page of items.
azure_queue_storage
Input
Dequeue objects from an Azure Storage Queue.
azure_table_storage
Input
:::caution BETA
batched
Input
Consumes data from a child input and applies a batching policy to the stream.
beanstalkd
Input
:::caution EXPERIMENTAL
broker
Input
Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel.
cassandra
Input
:::caution EXPERIMENTAL
cockroachdb_changefeed
Input
:::caution EXPERIMENTAL
csv
Input
Reads one or more CSV files as structured records following the format described in RFC 4180.
cypher
Input
:::caution EXPERIMENTAL
discord
Input
:::caution EXPERIMENTAL
dynamic
Input
A special broker type where the inputs are identified by unique labels and can be created, changed and removed during runtime via a REST HTTP interfac...
etcd
Input
:::caution BETA
file
Input
Consumes data from files on disk, emitting messages according to a chosen codec.
gcp_bigquery_select
Input
Executes a `SELECT` query against BigQuery and creates a message for each row received.
gcp_cloud_storage
Input
Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix.
gcp_pubsub
Input
Consumes messages from a GCP Cloud Pub/Sub subscription.
gcp_spanner_cdc
Input
:::caution BETA
generate
Input
Generates messages at a given interval using a Bloblang mapping executed without a context. This allows you to generate messages for testing your pipe...
hdfs
Input
Reads files from a HDFS directory, where each discrete file will be consumed as a single message payload.
http_client
Input
Connects to a server and continuously performs requests for a single message.
http_server
Input
Receive messages POSTed over HTTP(S). HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified.
inproc
Input
input:
kafka
Input
Connects to Kafka brokers and consumes one or more topics.
kafka_franz
Input
A Kafka input using the Franz Kafka client library.
mongodb
Input
Executes a query and creates a message for each document received.
mqtt
Input
Subscribe to topics on MQTT brokers.
nanomsg
Input
Consumes messages via Nanomsg sockets (scalability protocols).
nats
Input
Subscribe to a NATS subject.
nats_jetstream
Input
Reads messages from NATS JetStream subjects.
nats_kv
Input
Watches for updates in a NATS key-value bucket.
nats_object_store
Input
:::caution EXPERIMENTAL
nsq
Input
Subscribe to an NSQ instance topic and channel.
parquet
Input
:::caution EXPERIMENTAL
pulsar
Input
Reads messages from an Apache Pulsar server.
read_until
Input
Reads messages from a child input until a consumed message passes a Bloblang query, at which point the input closes. It is also possible to configure ...
redis_list
Input
Pops messages from the beginning of a Redis list using the BLPop command.
redis_pubsub
Input
Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands.
redis_scan
Input
Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands.
redis_streams
Input
Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The `client_id` should be unique for each consumer of a group.
resource
Input
Resource is an input type that channels messages from a resource input, identified by its name.
s2
Input
:::caution BETA
sequence
Input
Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next, and ...
sftp
Input
:::caution BETA
socket
Input
Connects to a tcp or unix socket and consumes a continuous stream of messages.
socket_server
Input
Creates a server that receives a stream of messages over a tcp, udp or unix socket.
sql_raw
Input
Executes a select query and creates a message for each row received.
sql_select
Input
Executes a select query and creates a message for each row received.
stdin
Input
Consumes data piped to stdin, chopping it into individual messages according to the specified scanner.
subprocess
Input
Executes a command, runs it as a subprocess, and consumes messages from it over stdout.
twitter_search
Input
:::caution EXPERIMENTAL
websocket
Input
Connects to a websocket server and continuously receives messages.
zmq4
Input
Consumes messages from a ZeroMQ socket.
zmq4n
Input
Consumes messages from a ZeroMQ socket.