Outputs

An output is a sink where data is sent after processing:

output:
  label: my_s3_output

  aws_s3:
    bucket: my-bucket
    path: 'data/${! timestamp_unix() }.json'

  # Optional list of processing steps before output
  processors:
    - mapping: |
        root = this.without("sensitive_field")

Brokering

Only one output is configured at the root of a pipeline config. However, the root output can be a broker which sends data to multiple destinations:

output:
  broker:
    pattern: fan_out
    outputs:
      - kafka:
          addresses: [ localhost:9092 ]
          topic: processed_data

      - aws_s3:
          bucket: archive-bucket
          path: 'archive/${! timestamp_unix() }.json'

Batching

Some outputs benefit from batching messages for better throughput. Configure batching with a batch policy:

output:
  kafka:
    addresses: [ localhost:9092 ]
    topic: events
    batching:
      count: 100
      period: 1s

Database Outputs

Write data to databases from your pipeline:

• sql_insert: Insert rows into database tables (MySQL, PostgreSQL, SQLite, etc.)
• sql_raw: Execute custom SQL statements

Batching is especially important for databases! Batch inserts can improve performance by 10-100x:

output:
  sql_insert:
    driver: postgres
    dsn: postgres://user:pass@localhost:5432/db
    table: events
    columns: ['id', 'data']
    args_mapping: 'root = [ this.id, this.data ]'
    batching:
      count: 500      # 50x faster than individual inserts
      period: 10s

See the Database Connectivity Guide for complete examples including:

• Writing to PostgreSQL and MySQL
• Using SQLite for edge storage
• Aggregating before cloud database writes
• Database replication patterns

Browse Outputs

Category:All CategoriesAIAWSAzureCachingCloudCompositionCoreDatabaseFilesFilterGCPHTTPIntegrationIoTLocalMappingMessagingNetworkNoSQLParsingRoutingSecurityServicesStorageStreamingTransformUtility

Status:All StatusStableBetaExperimental

Showing 55 of 184 components

aws_dynamodb

Output

Inserts items into or deletes items from a DynamoDB table.

ServicesAWSCloud+2

aws_kinesis

Output

Sends messages to a Kinesis stream.

ServicesAWSCloud

aws_kinesis_firehose

Output

Sends messages to a Kinesis Firehose delivery stream.

ServicesAWSCloud

aws_s3

Output

Sends message parts as objects to an Amazon S3 bucket. Each object is uploaded with the path specified with the `path` field.

ServicesAWSCloud+1

aws_sns

Output

Sends messages to an AWS SNS topic.

ServicesAWSCloud

aws_sqs

Output

Sends messages to an SQS queue.

ServicesAWSCloud

azure_blob_storage

Output

Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the `container` fi...

ServicesAzureCloud+1

azure_cosmosdb

Output

Creates or updates messages as JSON documents in Azure CosmosDB.

AzureCloud

azure_data_lake_gen2

Output

Sends message parts as files to an Azure Data Lake Gen2 file system.

ServicesAzureCloud+1

azure_queue_storage

Output

Sends messages to an Azure Storage Queue.

ServicesAzureCloud

azure_table_storage

Output

Stores messages in an Azure Table Storage table.

ServicesAzureCloud

broker

Output

Allows you to route messages to multiple child outputs using a range of brokering patterns.

Utility

cache

Output

Stores each message in a cache.

Services

drop

Output

Drops all messages.

Utility

drop_on

Output

Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead o...

Utility

dynamic

Output

A special broker type where the outputs are identified by unique labels and can be created, changed and removed during runtime via a REST API.

Utility

fallback

Output

Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the li...

Utility

file

Output

Writes messages to files on disk based on a chosen codec.

LocalFiles

gcp_bigquery

Output

Sends messages as new rows to a Google Cloud BigQuery table.

GCPServicesCloud

gcp_cloud_storage

Output

Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the `path` field.

ServicesGCPCloud

gcp_pubsub

Output

Sends messages to a GCP Cloud Pub/Sub topic. Metadata from messages are sent as attributes.

ServicesGCPCloud

http_client

Output

Sends messages to an HTTP server.

NetworkHTTP

http_server

Output

Sets up an HTTP server that will send messages over HTTP(S) GET requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert fil...

NetworkHTTP

inproc

Output

output:

Utility

kafka

Output

The kafka output type writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input.

ServicesMessagingStreaming

kafka_franz

Output

A Kafka output using the Franz Kafka client library.

ServicesMessagingStreaming

mqtt

Output

Pushes messages to an MQTT broker.

ServicesMessagingIoT

nats

Output

Publish to an NATS subject.

ServicesMessagingStreaming

nats_jetstream

Output

Write messages to a NATS JetStream subject.

ServicesMessagingStreaming

nats_kv

Output

Put messages in a NATS key-value bucket.

ServicesMessagingStreaming

nats_stream

Output

Publish messages to a NATS Stream subject.

ServicesMessagingStreaming

opensearch

Output

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

Services

qdrant

Output

Adds items to a Qdrant collection.

ServicesDatabase

redis_hash

Output

Sets Redis hash objects using the HMSET command.

ServicesMessagingCaching

redis_list

Output

Pushes messages onto the end of a Redis list (which is created if it doesn't already exist) using the RPUSH command.

ServicesMessagingCaching

redis_pubsub

Output

Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received.

ServicesMessagingCaching

redis_streams

Output

Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn't already exist) using the XADD command.

ServicesMessagingCaching

redpanda

Output

Sends message data to Kafka brokers and waits for acknowledgement before propagating any acknowledgements back to the input.

ServicesMessagingStreaming

reject

Output

Rejects all messages, treating them as though the output destination failed to publish them.

Utility

reject_errored

Output

Rejects messages that have failed their processing steps, resulting in nack behaviour at the input level, otherwise sends them to a child output.

Utility

resource

Output

Resource is an output type that channels messages to a resource output, identified by its name.

Utility

retry

Output

Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or m...

Utility

schema_registry

Output

Publishes schemas to a schema registry.

ServicesMessagingStreaming

snowflake_put

Output

Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables.

Services

snowflake_streaming

Output

Allows Snowflake to ingest data from your data pipeline using Snowpipe Streaming.

ServicesDatabase

socket

Output

Connects to a (tcp/udp/unix) server and sends a continuous stream of data, dividing messages according to the specified codec.

Network

splunk_hec

Output

Writes messages to a Splunk HTTP Endpoint Collector.

Services

sql

Output

Executes an arbitrary SQL query for each message.

ServicesDatabase

deprecated

sql_insert

Output

Inserts a row into an SQL database for each message.

ServicesDatabase

sql_raw

Output

Executes an arbitrary SQL query for each message.

ServicesDatabase

stdout

Output

Prints messages to stdout as a continuous stream of data.

Local

subprocess

Output

Executes a command, runs it as a subprocess, and writes messages to it over stdin.

Utility

switch

Output

The switch output type allows you to route messages to different outputs based on their contents.

UtilityTransformRouting

sync_response

Output

Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type.

Utility

websocket

Output

Sends messages to an HTTP server via a websocket connection.

NetworkStreaming