Skip to main content

Outputs

An output is a sink where data is sent after processing:

output:
  label: my_s3_output

  aws_s3:
    bucket: my-bucket
    path: 'data/${! timestamp_unix() }.json'

  # Optional list of processing steps before output
  processors:
    - mapping: |
        root = this.without("sensitive_field")

Brokering​

Only one output is configured at the root of a pipeline config. However, the root output can be a broker which sends data to multiple destinations:

output:
  broker:
    pattern: fan_out
    outputs:
      - kafka:
          addresses: [ localhost:9092 ]
          topic: processed_data

      - aws_s3:
          bucket: archive-bucket
          path: 'archive/${! timestamp_unix() }.json'

Batching​

Some outputs benefit from batching messages for better throughput. Configure batching with a batch policy:

output:
  kafka:
    addresses: [ localhost:9092 ]
    topic: events
    batching:
      count: 100
      period: 1s

Database Outputs​

Write data to databases from your pipeline:

  • sql_insert: Insert rows into database tables (MySQL, PostgreSQL, SQLite, etc.)
  • sql_raw: Execute custom SQL statements

Batching is especially important for databases! Batch inserts can improve performance by 10-100x:

output:
  sql_insert:
    driver: postgres
    dsn: postgres://user:pass@localhost:5432/db
    table: events
    columns: ['id', 'data']
    args_mapping: 'root = [ this.id, this.data ]'
    batching:
      count: 500      # 50x faster than individual inserts
      period: 10s

See the Database Connectivity Guide for complete examples including:

  • Writing to PostgreSQL and MySQL
  • Using SQLite for edge storage
  • Aggregating before cloud database writes
  • Database replication patterns

Browse Outputs​

Showing 55 of 185 components
aws_dynamodb
Output
Inserts items into a DynamoDB table.
ServicesAWS
aws_kinesis
Output
Sends messages to a Kinesis stream.
ServicesAWS
aws_kinesis_firehose
Output
Sends messages to a Kinesis Firehose delivery stream.
ServicesAWS
aws_s3
Output
Sends message parts as objects to an Amazon S3 bucket. Each object is uploaded with the path specified with the `path` field.
ServicesAWS
aws_sns
Output
Sends messages to an AWS SNS topic.
ServicesAWS
aws_sqs
Output
Sends messages to an SQS queue.
ServicesAWS
azure_blob_storage
Output
Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the `container` fi...
ServicesAzure
azure_cosmosdb
Output
Creates or updates messages as JSON documents in [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction).
Azure
azure_data_lake_gen2
Output
Sends message parts as files to an Azure Data Lake Gen2 filesystem. Each file is uploaded with the filename specified with the `path` field.
ServicesAzure
azure_queue_storage
Output
Sends messages to an Azure Storage Queue.
ServicesAzure
azure_table_storage
Output
Stores messages in an Azure Table Storage table.
ServicesAzure
broker
Output
Allows you to route messages to multiple child outputs using a range of brokering patterns.
Utility
cache
Output
Stores each message in a [cache](/components/caches/).
Services
drop
Output
Drops all messages.
Utility
drop_on
Output
Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead o...
Utility
dynamic
Output
A special broker type where the outputs are identified by unique labels and can be created, changed and removed during runtime via a REST API.
Utility
fallback
Output
Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the li...
Utility
file
Output
Writes messages to files on disk based on a chosen codec.
Local
gcp_bigquery
Output
Sends messages as new rows to a Google Cloud BigQuery table.
GCPServices
gcp_cloud_storage
Output
Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the `path` field.
ServicesGCP
gcp_pubsub
Output
Sends messages to a GCP Cloud Pub/Sub topic. [Metadata](/guides/pipelines/metadata) from messages are sent as attributes.
ServicesGCP
http_client
Output
Sends messages to an HTTP server.
Network
http_server
Output
Sets up an HTTP server that will send messages over HTTP(S) GET requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert fil...
Network
inproc
Output
Utility
kafka
Output
The kafka output type writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input.
Services
kafka_franz
Output
A Kafka output using the [Franz Kafka client library](https://github.com/twmb/franz-go).
Services
mqtt
Output
Pushes messages to an MQTT broker.
Services
nats
Output
Publish to an NATS subject.
Services
nats_jetstream
Output
Write messages to a NATS JetStream subject.
Services
nats_kv
Output
Put messages in a NATS key-value bucket.
Services
nats_stream
Output
Publish to a NATS Stream subject.
Services
opcua
Output
OPC UA output plugin
opensearch
Output
Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.
Services
qdrant
Output
Adds items to a [Qdrant](https://qdrant.tech/) collection
AI
redis_hash
Output
Sets Redis hash objects using the HSET command.
Services
redis_list
Output
Pushes messages onto the end of a Redis list (which is created if it doesn't already exist) using the RPUSH command.
Services
redis_pubsub
Output
Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received.
Services
redis_streams
Output
Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn't already exist) using the XADD command.
Services
redpanda
Output
A Kafka output using the [Franz Kafka client library](https://github.com/twmb/franz-go).
Services
reject
Output
Rejects all messages, treating them as though the output destination failed to publish them.
Utility
reject_errored
Output
Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output.
Utility
resource
Output
Resource is an output type that channels messages to a resource output, identified by its name.
Utility
retry
Output
Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or m...
Utility
schema_registry
Output
Publishes schemas to SchemaRegistry.
Integration
snowflake_put
Output
Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables.
Services
snowflake_streaming
Output
Ingest data into Snowflake using Snowpipe Streaming.
Services
socket
Output
Connects to a (tcp/udp/unix) server and sends a continuous stream of data, dividing messages according to the specified codec.
Network
sql
Output
Executes an arbitrary SQL query for each message.
Services
deprecated
sql_insert
Output
Inserts a row into an SQL database for each message.
Services
sql_raw
Output
Executes an arbitrary SQL query for each message.
Services
stdout
Output
Prints messages to stdout as a continuous stream of data.
Local
subprocess
Output
Executes a command, runs it as a subprocess, and writes messages to it over stdin.
Utility
beta
switch
Output
The switch output type allows you to route messages to different outputs based on their contents.
Utility
sync_response
Output
Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type.
Utility
websocket
Output
Sends messages to an HTTP server via a websocket connection.
Network