Outputs

An output is a sink where data is sent after processing:

output:
  label: my_s3_output

  aws_s3:
    bucket: my-bucket
    path: 'data/${! timestamp_unix() }.json'

  # Optional list of processing steps before output
  processors:
    - mapping: |
        root = this.without("sensitive_field")

Brokering

Only one output is configured at the root of a pipeline config. However, the root output can be a broker which sends data to multiple destinations:

output:
  broker:
    pattern: fan_out
    outputs:
      - kafka:
          addresses: [ localhost:9092 ]
          topic: processed_data

      - aws_s3:
          bucket: archive-bucket
          path: 'archive/${! timestamp_unix() }.json'

Batching

Some outputs benefit from batching messages for better throughput. Configure batching with a batch policy:

output:
  kafka:
    addresses: [ localhost:9092 ]
    topic: events
    batching:
      count: 100
      period: 1s

Database Outputs

Write data to databases from your pipeline:

• sql_insert: Insert rows into database tables (MySQL, PostgreSQL, SQLite, etc.)
• sql_raw: Execute custom SQL statements
• mongodb: Write to MongoDB collections
• cassandra: Write to Cassandra tables

Batching is especially important for databases! Batch inserts can improve performance by 10-100x:

output:
  sql_insert:
    driver: postgres
    dsn: postgres://user:pass@localhost:5432/db
    table: events
    columns: ['id', 'data']
    args_mapping: 'root = [ this.id, this.data ]'
    batching:
      count: 500      # 50x faster than individual inserts
      period: 10s

See the Database Connectivity Guide for complete examples including:

• Writing to PostgreSQL and MySQL
• Using SQLite for edge storage
• Aggregating before cloud database writes
• Database replication patterns

Browse Outputs

Category:All CategoriesAWSAzureCachingCloudCompositionCoreDatabaseFilesGCPHTTPIntegrationIoTLocalMachine LearningMappingMessagingNLPNetworkNoSQLObservabilityParsingRoutingSearchServicesSocialStorageStreamingTransformUtility

Status:All StatusStableBetaExperimental

Showing 68 of 200 components

amqp_0_9

Output

Sends messages to an AMQP (0.91) exchange. AMQP is a messaging protocol used by various message brokers, including RabbitMQ.Connects to an AMQP (0.91)...

ServicesMessaging

amqp_1

Output

Sends messages to an AMQP (1.0) server.

ServicesMessaging

aws_dynamodb

Output

Inserts items into or deletes items from a DynamoDB table.

ServicesAWSCloud+2

aws_kinesis

Output

Sends messages to a Kinesis stream.

ServicesAWSCloud

aws_kinesis_firehose

Output

Sends messages to a Kinesis Firehose delivery stream.

ServicesAWSCloud

aws_s3

Output

Sends message parts as objects to an Amazon S3 bucket. Each object is uploaded with the path specified with the `path` field.

ServicesAWSCloud+1

aws_sns

Output

Sends messages to an AWS SNS topic.

ServicesAWSCloud

aws_sqs

Output

Sends messages to an SQS queue.

ServicesAWSCloud

azure_blob_storage

Output

Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the `container` fi...

ServicesAzureCloud+1

azure_cosmosdb

Output

Creates or updates messages as JSON documents in Azure CosmosDB.

AzureCloud

azure_queue_storage

Output

Sends messages to an Azure Storage Queue.

ServicesAzureCloud

azure_table_storage

Output

:::caution BETA

ServicesAzureCloud

beta

beanstalkd

Output

:::caution EXPERIMENTAL

Services

experimental

broker

Output

Allows you to route messages to multiple child outputs using a range of brokering patterns.

Utility

cache

Output

Stores each message in a cache.

Services

cassandra

Output

Runs a query against a Cassandra database for each message in order to insert data.

DatabaseNoSQL

cypher

Output

:::caution EXPERIMENTAL

Services

experimental

discord

Output

:::caution EXPERIMENTAL

ServicesSocial

experimental

drop

Output

Drops all messages.

Utility

drop_on

Output

Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead o...

Utility

dynamic

Output

A special broker type where the outputs are identified by unique labels and can be created, changed and removed during runtime via a REST API.

Utility

elasticsearch_v2

Output

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

ServicesDatabaseSearch+1

fallback

Output

Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the li...

Utility

file

Output

Writes messages to files on disk based on a chosen codec.

LocalFiles

gcp_bigquery

Output

Sends messages as new rows to a Google Cloud BigQuery table.

GCPServicesCloud

gcp_bigquery_write_api

Output

Sends messages as new rows to a Google Cloud BigQuery table using the BigQuery Storage Write API.

GCPServicesCloud

gcp_cloud_storage

Output

Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the `path` field.

ServicesGCPCloud

gcp_pubsub

Output

Sends messages to a GCP Cloud Pub/Sub topic. Metadata from messages are sent as attributes.

ServicesGCPCloud

hdfs

Output

Sends message parts as files to a HDFS directory.

Services

http_client

Output

Sends messages to an HTTP server.

NetworkHTTP

http_server

Output

Sets up an HTTP server that will send messages over HTTP(S) GET requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert fil...

NetworkHTTP

inproc

Output

output:

Utility

kafka

Output

The kafka output type writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input.

ServicesMessagingStreaming

kafka_franz

Output

A Kafka output using the Franz Kafka client library.

ServicesMessagingStreaming

mongodb

Output

Inserts items into a MongoDB collection.

ServicesDatabaseNoSQL

mqtt

Output

Pushes messages to an MQTT broker.

ServicesMessagingIoT

nanomsg

Output

Send messages over a Nanomsg socket.

Network

nats

Output

Publish to an NATS subject.

ServicesMessagingStreaming

nats_jetstream

Output

Write messages to a NATS JetStream subject.

ServicesMessagingStreaming

nats_kv

Output

Put messages in a NATS key-value bucket.

ServicesMessagingStreaming

nats_object_store

Output

:::caution EXPERIMENTAL

ServicesMessagingStreaming

experimental

nsq

Output

Publish to an NSQ topic.

Services

opensearch

Output

Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping.

Services

pulsar

Output

Write messages to an Apache Pulsar server.

ServicesMessagingStreaming

pusher

Output

:::caution EXPERIMENTAL

Services

experimental

questdb

Output

:::caution EXPERIMENTAL

Services

experimental

redis_hash

Output

Sets Redis hash objects using the HMSET command.

ServicesMessagingCaching

redis_list

Output

Pushes messages onto the end of a Redis list (which is created if it doesn't already exist) using the RPUSH command.

ServicesMessagingCaching

redis_pubsub

Output

Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received.

ServicesMessagingCaching

redis_streams

Output

Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn't already exist) using the XADD command.

ServicesMessagingCaching

reject

Output

Rejects all messages, treating them as though the output destination failed to publish them.

Utility

reject_errored

Output

Rejects messages that have failed their processing steps, resulting in nack behaviour at the input level, otherwise sends them to a child output.

Utility

resource

Output

Resource is an output type that channels messages to a resource output, identified by its name.

Utility

retry

Output

Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or m...

Utility

s2

Output

:::caution BETA

Services

beta

sftp

Output

:::caution BETA

NetworkFiles

beta

snowflake_put

Output

Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables.

Services

socket

Output

Connects to a (tcp/udp/unix) server and sends a continuous stream of data, dividing messages according to the specified codec.

Network

splunk_hec

Output

Writes messages to a Splunk HTTP Endpoint Collector.

Services

sql_insert

Output

Inserts a row into an SQL database for each message.

Services

sql_raw

Output

Executes an arbitrary SQL query for each message.

Services

stdout

Output

Prints messages to stdout as a continuous stream of data.

Local

subprocess

Output

Executes a command, runs it as a subprocess, and writes messages to it over stdin.

Utility

switch

Output

The switch output type allows you to route messages to different outputs based on their contents.

UtilityTransformRouting

sync_response

Output

Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type.

Utility

websocket

Output

Sends messages to an HTTP server via a websocket connection.

NetworkStreaming

zmq4

Output

Writes messages to a ZeroMQ socket.

Network

zmq4n

Output

Writes messages to a ZeroMQ socket.

Network