Skip to main content

parquet

Converts batches of documents to or from Parquet files.

This processor performs bidirectional conversion between JSON documents and the Parquet file format. Use the operator field to choose the direction: from_json compresses a batch of JSON documents into a single Parquet file, and to_json expands a Parquet file into a batch of JSON messages.

# Config fields, showing default values
processor:
  label: ""
  parquet:
    operator: "" # No default (required)
    compression: snappy
    schema: ""
    schema_file: ""

Examples​

Compress a batch of JSON documents into a single Parquet file using an inline schema and Snappy compression.

processor:
  parquet:
    operator: from_json
    compression: snappy
    schema: |-
      {
        "Tag": "name=root, repetitiontype=REQUIRED",
        "Fields": [
          {"Tag":"name=name,inname=NameIn,type=BYTE_ARRAY,convertedtype=UTF8, repetitiontype=REQUIRED"},
          {"Tag":"name=age,inname=Age,type=INT32,repetitiontype=REQUIRED"}
        ]
      }

Fields​

operator​

Determines the direction of conversion. Use from_json to compress a batch of JSON documents into a single Parquet file, or to_json to expand a Parquet file into a batch of JSON messages.

Type: string

compression​

The type of compression to use when writing Parquet files. Supported values are uncompressed, snappy, gzip, lz4, and zstd. This field is ignored when the operator is to_json.

Type: string
Default: "snappy"

schema​

An inline JSON document describing the tags and fields of the Parquet schema to use. Either schema or schema_file must be set when creating Parquet files with the from_json operator.

Type: string

schema_file​

A path to a file containing a JSON document that describes the tags and fields of the Parquet schema to use. Either schema_file or schema must be set when creating Parquet files with the from_json operator.

Type: string