Skip to main content

Data Routing & Distribution

Route data intelligently based on content, priority, or destination requirements. These guides show you how to send data to multiple systems, implement priority queues, protect downstream services, and split messages for different processing paths.

Guides in This Category​

Send Data to Multiple Destinations​

Problem: You need to send the same data to multiple systems simultaneously—real-time processing AND long-term storage.

Solution: Use broker output with fan_out pattern to send data to Kafka, S3, and Elasticsearch with independent retry logic per destination.

Key Benefits: Real-time + batch processing, independent failure handling, parallel delivery

Read the guide: Send Data to Multiple Destinations →

Route Data Based on Content​

Problem: Different types of data need different handling—critical errors to one destination, info logs to another.

Solution: Use switch processor for conditional routing based on severity, region, event type, or any field value.

Key Benefits: 66% bandwidth reduction, intelligent filtering, priority handling

Read the guide: Route Data Based on Content →

Implement Content-Based Splitting​

Problem: Batch messages or arrays need to be split into individual events for per-message processing.

Solution: Use split processor to break JSON arrays, CSV batches, or multiline logs into individual messages.

Key Benefits: Enable per-message routing, parallel processing, granular error handling

Read the guide: Implement Content-Based Splitting →

Create Priority Queues​

Problem: Critical messages get stuck behind low-priority data, violating SLAs.

Solution: Route messages to separate queues based on priority levels with weighted processing.

Key Benefits: SLA compliance, fair scheduling, prevent starvation, 93% cost reduction for high-priority data

Read the guide: Create Priority Queues →

Implement Circuit Breakers​

Problem: Failing downstream systems cause cascading failures and resource exhaustion.

Solution: Implement circuit breaker pattern with retry limits, fallback destinations, and automatic recovery.

Key Benefits: Prevent cascading failures, automatic recovery, resource protection, graceful degradation

Read the guide: Implement Circuit Breakers →

Common Routing Patterns​

Pattern 1: Fan-Out (One-to-Many)​

         ┌─→ Kafka (real-time)
Input ───┼─→ S3 (archive)
         └─→ Elasticsearch (search)

Use when: Same data needs multiple destinations Guide: Send Data to Multiple Destinations

Pattern 2: Content-Based Routing​

         ┌─→ [severity=error] → Kafka Topic: errors
Input ───┼─→ [severity=warn]  → Kafka Topic: warnings
         └─→ [severity=info]  → Local File

Use when: Different data types need different handling Guide: Route Data Based on Content

Pattern 3: Priority Queuing​

         ┌─→ [priority=critical] → Fast Queue (10 workers)
Input ───┼─→ [priority=high]     → Medium Queue (5 workers)
         └─→ [priority=normal]   → Slow Queue (2 workers)

Use when: SLA requirements vary by message type Guide: Create Priority Queues

Pattern 4: Circuit Breaker with Fallback​

Input ─→ Circuit Breaker ─┬─→ [Closed] → Destination
                          └─→ [Open]   → Dead Letter Queue

Use when: Downstream systems have availability issues Guide: Implement Circuit Breakers

Pattern 5: Split and Route​

Batch Input ─→ Split ─┬─→ [type=A] → Destination 1
                      ├─→ [type=B] → Destination 2
                      └─→ [type=C] → Destination 3

Use when: Batches contain different message types Guide: Implement Content-Based Splitting