Rate Limits

A rate limiter restricts the rate at which messages can pass through your pipeline. Rate limiters are configured as a named resource:

rate_limit_resources:
  - label: api_limiter
    local:
      count: 100
      interval: 1s

And then any components that use rate limiters have a field resource that specifies the rate limit resource:

pipeline:
  processors:
    - rate_limit:
        resource: api_limiter
    - http:
        url: https://api.example.com/endpoint
        verb: POST

Rate limiters are useful for:

• Respecting third-party API rate limits
• Preventing downstream system overload
• Controlling costs for metered services
• Ensuring fair resource sharing

For distributed rate limiting across multiple pipeline instances, use the redis rate limiter.

You can find out more about resources in the resources guide.