Skip to main content

gcp_vertex_ai_chat

Generates responses to messages in a chat conversation, using the Vertex AI API.

This processor sends prompts to a chosen large language model (LLM) and generates text from the responses via the Vertex AI API. You can supply the prompt directly, include prior conversation history, attach additional data such as an image, and optionally allow the model to invoke tools.

# Config fields, showing default values
processor:
  label: ""
  gcp_vertex_ai_chat:
    project: "" # No default (required)
    credentials_json: ""
    location: "" # No default (required)
    model: "" # No default (required)
    prompt: ""
    system_prompt: ""
    history: ""
    attachment: ""
    temperature: 0
    max_tokens: 0
    response_format: text
    top_p: 0
    top_k: 0
    stop: []
    presence_penalty: 0
    frequency_penalty: 0
    max_tool_calls: 10
    tools:
      - name: "" # No default (required)
        description: "" # No default (required)
        parameters:
          properties: {} # No default (required)
          required: []
        processors: []

Examples​

Send each message to a model with a system prompt and replace it with the generated response.

processor:
  gcp_vertex_ai_chat:
    project: my-gcp-project
    location: us-central1
    model: gemini-1.5-pro
    system_prompt: "You are a helpful assistant that summarizes log data in one sentence."

Fields​

project​

The ID of your Google Cloud project.

Type: string

credentials_json​

Set your Google Service Account Credentials as JSON. :::warning Secret This field contains sensitive information that usually shouldn't be added to a config directly, read our secrets page for more info. :::

Type: string
Default: ""

location​

The location of the Vertex AI large language model (LLM) that you want to use. For base models you can omit this field.

Type: string

model​

The name of the LLM to use. For a full list of models, see the Vertex AI Model Garden.

Type: string

prompt​

The prompt you want to generate a response for. By default, the processor submits the entire payload of each message as a string.

Type: string

system_prompt​

The system prompt to submit along with the prompt.

Type: string

history​

Historical messages to include in the chat request.

Type: string

attachment​

Additional data, such as an image, to send along with the prompt.

Type: string

temperature​

Controls the randomness of the model's predictions. Higher values produce more random output.

Type: float

max_tokens​

The maximum number of output tokens to generate per response.

Type: int

response_format​

The format of the generated response. Set to text for plain text or json for a JSON object.

Type: string
Default: "text"

top_p​

Enables nucleus sampling. The model considers only the smallest set of tokens whose cumulative probability exceeds this value.

Type: float

top_k​

Enables top-k sampling. The model samples from the most likely tokens up to this count.

Type: float

stop​

Sets the stop sequences to use. The model stops generating further tokens when it produces one of these sequences.

Type: array

presence_penalty​

Positive values penalize new tokens if they already appear in the generated text.

Type: float

frequency_penalty​

Positive values penalize new tokens based on their existing frequency in the generated text.

Type: float

max_tool_calls​

The maximum number of sequential tool calls the model may make when responding to a prompt.

Type: int
Default: 10

tools​

The tools to allow the LLM to invoke. This extends the capabilities of the LLM to the tools that you define. To use no external tools, provide an empty array.

Type: array
Default: []

tools.name​

The name of this tool.

Type: string

tools.description​

A description of this tool. The model uses this to decide when the tool should be invoked.

Type: string

tools.parameters​

The parameters the model needs to provide in order to invoke this tool.

Type: object

tools.parameters.properties​

The properties for the tool's input data.

Type: object

tools.parameters.required​

The list of required parameters.

Type: array
Default: []

tools.processors​

The pipeline of processors to execute when the LLM uses this tool.

Type: array