Skip to main content

azure_data_lake_gen2

Sends message parts as files to an Azure Data Lake Gen2 file system.

Each file is uploaded to the configured file system with the file name given by the path field, which supports interpolation functions so you can derive names from message content or metadata. Multiple authentication methods are supported, and messages can be processed in parallel to improve throughput.

# Config fields, showing default values
output:
  label: ""
  azure_data_lake_gen2:
    storage_account: ""
    storage_access_key: ""
    storage_connection_string: ""
    storage_sas_token: ""
    filesystem: "" # No default (required)
    path: ${!counter()}-${!timestamp_unix_nano()}.txt
    max_in_flight: 64

Examples​

Authenticate with a connection string and upload each message to a file system whose name includes the current year.

output:
  azure_data_lake_gen2:
    storage_connection_string: "DefaultEndpointsProtocol=https;AccountName=example;AccountKey=...;EndpointSuffix=core.windows.net"
    filesystem: messages-${!timestamp("2006")}
    path: ${!counter()}-${!timestamp_unix_nano()}.json

Fields​

storage_account​

The storage account to access. This field is ignored if storage_connection_string is set.

Type: string
Default: ""

storage_access_key​

The storage account access key. This field is ignored if storage_connection_string is set.

Type: string
Default: ""

storage_connection_string​

A storage account connection string. This field is required if storage_account and storage_access_key / storage_sas_token are not set.

Type: string
Default: ""

storage_sas_token​

The storage account SAS token. This field is ignored if storage_connection_string or storage_access_key are set.

Type: string
Default: ""

filesystem​

The name of the data lake storage file system you want to upload messages to. This field supports interpolation functions.

Type: string

path​

The path (file name) of each message to upload. This field supports interpolation functions.

Type: string
Default: "${!counter()}-${!timestamp_unix_nano()}.txt"

max_in_flight​

The maximum number of messages to have in flight at a given time. Increase this to improve throughput.

Type: int
Default: 64