Skip to main content

gcp_bigquery_select processor

Executes a SELECT query against BigQuery and replaces messages with the rows returned.

# Config fields, showing default values
pipeline:
  processors:
    - label: ""
      gcp_bigquery_select:
        project: "" # No default (required)
        credentials_json: ""
        table: "" # No default (required)
        columns: [] # No default (required)
        where: "" # No default (optional)
        job_labels: {}
        args_mapping: "" # No default (optional)
        prefix: "" # No default (optional)
        suffix: "" # No default (optional)

Examples​

Word count​

Given a stream of English terms, enrich the messages with the word count from Shakespeare's public works:

pipeline:
  processors:
    - branch:
        processors:
          - gcp_bigquery_select:
              project: test-project
              table: bigquery-public-data.samples.shakespeare
              columns:
                - word
                - sum(word_count) as total_count
              where: word = ?
              suffix: |
                GROUP BY word
                ORDER BY total_count DESC
                LIMIT 10
              args_mapping: root = [ this.term ]
        result_map: |
          root.count = this.get("0.total_count")

Fields​

project​

GCP project where the query job will execute.

Type: string

credentials_json​

An optional field to set Google Service Account Credentials json.

Secret

This field contains sensitive information. Use a secret reference rather than a literal value.

Type: string
Default: ""

table​

Fully-qualified BigQuery table name to query.

Type: string

columns​

A list of columns to query.

Type: array of string

where​

An optional where clause to add. Placeholder arguments are populated with the args_mapping field. Placeholders should always be question marks (?).

Type: string

job_labels​

A list of labels to add to the query job.

Type: map of string
Default: {}

args_mapping​

An optional Bloblang mapping which should evaluate to an array of values matching in size to the number of placeholder arguments in the field where.

Type: string

prefix​

An optional prefix to prepend to the select query (before SELECT).

Type: string

suffix​

An optional suffix to append to the select query.

Type: string