protobuf processor

Performs conversions to or from a protobuf message. This processor uses reflection, meaning conversions can be made directly from the target .proto files.

# Config fields, showing default values
pipeline:
  processors:
    - label: ""
      protobuf:
        operator: "" # No default (required)
        message: "" # No default (required)
        discard_unknown: false
        use_proto_names: false
        import_paths: []
        use_enum_numbers: false
        bsr: []

The main functionality of this processor is to map to and from JSON documents, you can read more about JSON mapping of protobuf messages here: https://developers.google.com/protocol-buffers/docs/proto3#json

Using reflection for processing protobuf messages in this way is less performant than generating and using native code. Therefore when performance is critical it is recommended that you use native plugins instead for processing protobuf messages, following the upstream engine's plugin example.

The processor will ignore any files that begin with a dot ("."g), a convention for hidden files, when loading protocol buffer definitions.

Operators

to_json

Converts protobuf messages into serialized proto3 JSON.

from_json

Attempts to create a target protobuf message from a serialized proto3 JSON.

decode

Converts protobuf messages into a generic structured message. This makes it easier to manipulate the contents of the document within Expanso Edge. This differs from to_json in the following ways:

• 64 bit numbers are not converted into strings
• Bytes and google.protobuf.Timestamp types are preserved (not encoded as strings unless serialized)

This operator is also considerably faster in scenario where you manipulate the data as the data does not need to be serialized then deserialized like with the to_json operator.

Examples

JSON to Protobuf using Schema from Disk

If we have the following protobuf definition within a directory called testing/schema:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of JSON documents of the form:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "[email protected]"
}

We can convert the documents into protobuf messages with the following config:

pipeline:
  processors:
    - protobuf:
        operator: from_json
        message: testing.Person
        import_paths: [ testing/schema ]

Protobuf to JSON using Schema from Disk

If we have the following protobuf definition within a directory called testing/schema:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of protobuf messages of the type Person, we could convert them into JSON documents of the format:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "[email protected]"
}

With the following config:

pipeline:
  processors:
    - protobuf:
        operator: to_json
        message: testing.Person
        import_paths: [ testing/schema ]

JSON to Protobuf using Buf Schema Registry

If we have the following protobuf definition within a BSR module hosted at buf.build/exampleco/mymodule:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of JSON documents of the form:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "[email protected]"
}

We can convert the documents into protobuf messages with the following config:

pipeline:
  processors:
    - protobuf:
        operator: from_json
        message: testing.Person
        bsr:
          - module: buf.build/exampleco/mymodule
            api_key: xxx

Protobuf to JSON using Buf Schema Registry

If we have the following protobuf definition within a BSR module hosted at buf.build/exampleco/mymodule:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of protobuf messages of the type Person, we could convert them into JSON documents of the format:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "[email protected]"
}

With the following config:

pipeline:
  processors:
    - protobuf:
        operator: to_json
        message: testing.Person
        bsr:
          - module: buf.build/exampleco/mymodule
            api_key: xxxx

Fields

operator

The operator to execute

Type: string

Options: to_json, from_json, decode

message

The fully qualified name of the protobuf message to convert to/from.

Type: string

discard_unknown

If true, the from_json operator discards fields that are unknown to the schema.

Type: bool
Default: false

use_proto_names

If true, the to_json or decode operator deserializes fields exactly as named in schema file.

Type: bool
Default: false

import_paths

A list of directories containing .proto files, including all definitions required for parsing the target message. If left empty the current directory is used. Each directory listed will be walked with all found .proto files imported. Either this field or bsr must be populated.

Type: array of string
Default: []

use_enum_numbers

If true, the to_json or decode operator deserializes enums as numerical values instead of string names.

Type: bool
Default: false

bsr

Buf Schema Registry configuration. Either this field or import_paths must be populated. Note that this field is an array, and multiple BSR configurations can be provided.

Type: array of object
Default: []

bsr[].module

Module to fetch from a Buf Schema Registry e.g. 'buf.build/exampleco/mymodule'.

Type: string

bsr[].url

Buf Schema Registry URL, leave blank to extract from module.

Type: string
Default: ""

bsr[].api_key

Buf Schema Registry server API key, can be left blank for a public registry.

Secret

This field contains sensitive information. Use a secret reference rather than a literal value.

Type: string
Default: ""

bsr[].version

Version to retrieve from the Buf Schema Registry, leave blank for latest.

Type: string
Default: ""