Skip to main content

ollama_chat

Generates responses to messages in a chat conversation using the Ollama API and external tools.

This processor sends a prompt to a large language model running on Ollama and replaces the message with the model's generated response, with support for external tools. By default it starts and runs a locally installed Ollama server, or you can point it at an existing server using the server_address field.

# Config fields, showing default values
processor:
  label: ""
  ollama_chat:
    model: "" # No default (required)
    prompt: ""
    system_prompt: ""
    history: ""
    image: ""
    response_format: text
    max_tokens: 0
    temperature: 0
    seed: 0
    num_keep: 0
    top_k: 0
    top_p: 0
    repeat_penalty: 0
    presence_penalty: 0
    frequency_penalty: 0
    stop: []
    max_tool_calls: 3
    tools:
      - name: ""
        description: ""
        parameters:
          properties: {}
          required: []
        processors: []
    save_prompt_metadata: false
    server_address: ""
    cache_directory: ""
    download_url: ""
    runner:
      context_size: 0
      batch_size: 0
      gpu_layers: 0
      threads: 0
      use_mmap: false

Examples​

Use a multimodal model to describe an image supplied in the message payload.

processor:
  ollama_chat:
    model: llava
    prompt: "Describe the following image."
    image: "root = content()"

Fields​

model​

The name of the Ollama LLM to use, for example llama3.1, gemma2, qwen2, or phi3.

Type: string

prompt​

The prompt you want to generate a response for. By default the processor submits the entire message payload as the prompt. This field supports interpolation functions.

Type: string
Default: ""

system_prompt​

The system prompt to submit to the Ollama LLM. This field supports interpolation functions.

Type: string
Default: ""

history​

Historical messages to include in the chat request for additional context. Use a Bloblang query to produce an array of objects in the form [{"role":"user","content":"<text>"}], where role is one of system, user, assistant, or tool and content is the message text.

Type: string
Default: ""

image​

An optional image to submit along with the prompt. The result of the Bloblang mapping must be a byte array.

Type: string
Default: ""

response_format​

The format of the response that the Ollama LLM generates. Set to text to return a plain text response, or json to constrain the output to valid JSON. When set to json, instruct the model to produce JSON in the prompt or system prompt.

Type: string
Default: text

max_tokens​

The maximum number of tokens to generate in the response.

Type: int

temperature​

The temperature used for sampling. Higher values produce more creative and varied responses, while lower values make the output more focused and deterministic.

Type: int

seed​

Sets the random number seed to use for generation. Setting this to a specific number makes the model generate the same response for the same prompt and parameters.

Type: int

num_keep​

Specifies the number of tokens from the initial prompt to retain when the model resets its internal context. By default the model retains 4 tokens; set this to -1 to keep all tokens from the initial prompt.

Type: int

top_k​

Reduces the probability of generating nonsense. A higher value, for example 100, produces more diverse answers, while a lower value, for example 10, is more conservative.

Type: int

top_p​

Works together with top_k. A higher value, for example 0.95, produces more diverse text, while a lower value, for example 0.5, generates more focused and conservative text.

Type: float

repeat_penalty​

Sets how strongly to penalize repetitions. A higher value, for example 1.5, penalizes repetitions more strongly, while a lower value, for example 0.9, is more lenient.

Type: float

presence_penalty​

Positive values penalize new tokens based on whether they have already appeared in the text, increasing the model's likelihood to talk about new topics.

Type: float

frequency_penalty​

Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

Type: float

stop​

A list of sequences where the model stops generating further tokens. When the model encounters one of these sequences it stops generating text and returns the response.

Type: array

max_tool_calls​

The maximum number of sequential calls the model can make to external tools while generating a single response.

Type: int
Default: 3

tools​

External tools that the model can invoke, such as functions or APIs, to help answer a prompt. Define each tool as a set of processors, and the model chooses when to call them. To use no external tools, provide an empty array.

Type: array
Default: []

tools.name​

The name of this tool.

Type: string

tools.description​

A description of what this tool does. The model uses this to decide whether the tool should be invoked.

Type: string

tools.parameters​

The parameters that the model must provide in order to invoke this tool.

Type: object

tools.parameters.properties​

The properties that define the tool's input parameters.

Type: object

tools.parameters.required​

The list of required parameter names for this tool.

Type: array
Default: []

tools.processors​

The pipeline of processors to execute when the model invokes this tool. The result of the pipeline is returned to the model.

Type: array

save_prompt_metadata​

When enabled, saves the prompt used for the request in the @prompt metadata field, and the system prompt in the @system_prompt metadata field when a system prompt is configured.

Type: bool
Default: false

server_address​

The address of the Ollama server to use. Leave blank to start and run a locally installed Ollama server.

Type: string

cache_directory​

If server_address is not set, the directory used to download the Ollama binary and cache models, for example /opt/cache/connect/ollama.

Type: string

download_url​

If server_address is not set, the URL to download the Ollama binary from. Defaults to the official Ollama GitHub release.

Type: string

runner​

Options for the model runner that are applied when the model is first loaded and run locally.

Type: object

runner.context_size​

Sets the size of the context window used to generate the next token. A larger context window uses more memory and takes longer to process.

Type: int

runner.batch_size​

The maximum number of requests to process in parallel.

Type: int

runner.gpu_layers​

The number of model layers to offload to the GPU. By default the runtime decides how many layers to offload based on the available memory.

Type: int

runner.threads​

The number of threads to use during response generation. By default the runtime uses the number of physical CPU cores.

Type: int

runner.use_mmap​

Maps the model into memory so the system loads only the necessary parts as needed. This is supported only on Unix systems.

Type: bool