# Use Cases Reference

> Auto-generated from docs.expanso.io documentation

## Use Cases

# Use Cases

Expanso Edge enables powerful data processing at the source, reducing costs, improving performance, and ensuring compliance. Explore the examples below to discover how edge processing can address your specific needs.

## Data Lineage

# Data Lineage

## Overview

Regulated, multi-region, and high-trust deployments need to prove what their data pipelines did — which inputs were read, which outputs were written, when the run started and ended, and whether it completed cleanly. Centralized lineage tools answer this only if every system contributing data feeds them events. When pipelines run at the edge — across thousands of nodes, in disconnected sites, under regional sovereignty rules — the lineage trail starts at the edge, not at the central catalog.

## Expanso's Approach to Data Lineage

Expanso Edge emits signed [OpenLineage](https://openlineage.io/) events natively, directly from the node where each pipeline ran. A lifecycle event fires at every transition (`START`, `COMPLETE`, `FAIL`, `ABORT`) and is delivered to any OpenLineage-compatible backend — [Marquez](https://marquezproject.ai/), data catalogs, or custom collectors — over HTTP or persisted to disk for air-gapped collection.

**Key capabilities:**

- **Native emission at the source.** Events fire from the edge agent running the pipeline, not from a centralized scrubbing job — so the lineage record matches the operational truth even when nodes are offline, regionally segregated, or running under data-sovereignty rules.
- **Signed events.** Every event carries an Ed25519 signature over the canonical event bytes. Downstream consumers verify both producer identity and payload integrity, supporting tamper-evident audit trails.
- **Schema-conformant.** Events validate against the OpenLineage 2.0.2 `RunEvent` schema and drop into Marquez or any OpenLineage-aware tool without translation.
- **Two transports.** HTTP delivery for connected sites; JSON-Lines file output for air-gapped collection, shipped onward by your existing log infrastructure.
- **Dataset extraction.** Input and output components are mapped to OpenLineage `Dataset`s automatically (Kafka topics, S3 buckets, files, SQL tables, HTTP endpoints).
- **Pipeline-safe.** Lineage emission is fully asynchronous; failures route to the node's health surface and never block, fail, or slow the pipeline.
- **Aligned message metadata.** The [`metadata` processor](/components/processors/metadata) stamps the same `run_id` on each message body, so per-event records correlate back to the lifecycle events emitted by the edge.

## Benefits of Edge-Emitted Lineage

**Audit & Compliance**

- Tamper-evident, signed event stream from every node — supports SOC 2, ISO 27001, regulated-industry audit requirements
- Lineage record produced where the data lives, not reconstructed after the fact
- Per-run identity, dataset references, and outcome facets satisfy the "who, what, when" questions in regulatory reviews
- File-transport mode lets air-gapped or sovereignty-bound sites produce lineage without phoning home

**Data Governance**

- Populate Marquez, DataHub, and other OpenLineage-aware catalogs without writing connectors per pipeline
- Discover which jobs read or write each dataset across a fleet, automatically
- Distinguish natural completions from operator interventions (`COMPLETE` vs `ABORT`) for incident reviews
- Custom `expanso_run` facet carries per-run counters (records in/out, bytes in/out, error count) for capacity and quality reporting

**Operational Resilience**

- Lineage delivery never blocks or fails the pipeline — drops are counted and visible, but do not corrupt runtime behavior
- HTTP failures surface in the node's `HealthTracker` for the affected execution, visible via `expanso-cli execution describe`
- A single, low-cardinality metric (`lineage_events_dropped_total`) drives alerts without overwhelming telemetry budgets
- File transport keeps lineage flowing even when the central backend is down

## Common Patterns

**Edge-First Audit Trail**

Configure every edge node to emit lineage to a regional collector with signed events. Auditors verify the per-run signature against each node's published key to prove that the record came from the producing node and was not altered downstream. See [Verify Lineage Event Signatures](/examples/verify-lineage-events) for the verification recipe.

**Air-Gapped Collection**

Use the file transport on disconnected nodes. A standard log-shipper (Fluent Bit, vector, your own script) reads the JSON-Lines file and forwards events to a regional collector when connectivity is available. Each line is a complete, schema-conformant event; ordering and replay are managed by the shipper.

```yaml
lineage:
  enabled: true
  transport: file
  file:
    path: /var/lib/expanso/lineage/events.jsonl
    rotation_size_mb: 64
```

**Correlated Body Metadata**

Pair the lineage emitter with the [`metadata` processor](/components/processors/metadata) so every message carries the same `run_id` that appears on the lifecycle event. Downstream stores can join records back to the lifecycle events without bespoke joins.

```yaml
pipeline:
  processors:
    - metadata:
        include: [core, pipeline]
        target: body
        format: nested
        body_key: lineage
```

**Regional Marquez per Sovereignty Zone**

Run one Marquez instance per region. Each edge node's lineage config points at its regional endpoint, so events stay within the sovereignty boundary while still feeding a centralized governance view through Marquez's existing federation patterns.

**Failure Forensics**

`FAIL` events carry an `errorMessage` facet with the classified pipeline error and stack trace. Catalogs that index lineage events become a queryable history of pipeline failures across the fleet — useful for postmortems and for spotting silent regressions in production.

## Example Use Cases

- **Regulated financial services** producing per-execution audit trails for every trade-data pipeline, with signed lineage events feeding a compliance catalog and a separate audit archive
- **Healthcare data platforms** demonstrating HIPAA "minimum necessary" by recording which datasets each pipeline read at the edge and what it wrote downstream
- **Multi-region SaaS** running Marquez per region for data sovereignty, with cross-region lineage roll-up through Marquez federation rather than centralizing the raw events
- **Industrial / IoT operators** with disconnected sites using the file transport to emit lineage locally and ship to a central collector when connectivity returns

## Getting Started

1. **Enable emission** — follow the [OpenLineage Emission how-to](/guides/pipelines/lineage). The minimal change is two lines of edge config plus a restart.
2. **Stand up Marquez locally** — the how-to includes a `docker-compose.marquez.yaml` for seeing your edge's events in a UI within minutes.
3. **Pair with body metadata** — add the [`metadata` processor](/components/processors/metadata) to your pipelines so per-message records carry the same `run_id`.
4. **Verify signatures downstream** — once events are flowing, plug in the [signature verification recipe](/examples/verify-lineage-events) at your collector or audit boundary.

## Related

- **[OpenLineage Emission how-to](/guides/pipelines/lineage)** — end-to-end walkthrough including Marquez setup.
- **[Lineage Configuration](/references/configuration/edge-config)** — full configuration reference.
- **[Verify Lineage Event Signatures](/examples/verify-lineage-events)** — Ed25519 verification with Go and Python recipes.
- **[`metadata` processor](/components/processors/metadata)** — attach OpenLineage-aligned identity to every message body.
- **[Sign Pipeline Messages](/examples/signing-and-verifying)** — per-message signing using the same edge identity.

## Distributed Analytics

# Distributed Analytics

## Overview

Organizations operating across multiple regions face the challenge of collecting, processing, and analyzing data while maintaining compliance with regional data sovereignty laws. Traditional approaches either sacrifice centralized analytics for compliance or risk violations by centralizing raw data. Edge processing enables both: regional data processing that meets compliance requirements while providing centralized analytics and insights.

## Expanso's Approach to Distributed Analytics

Expanso Edge processes data regionally at the source, applying local transformations, aggregations, and compliance rules before sending analytics-ready data to centralized systems. Each region operates independently while contributing to a unified analytical view.

**Key capabilities:**

- **Regional Processing**: Run complete analytics pipelines in each geographic region, ensuring data is processed locally before any cross-border transmission.
- **Compliance-First Aggregation**: Apply regional privacy rules, PII redaction, and data minimization at the edge before creating analytics datasets.
- **Centralized Analytics**: Aggregate pre-processed, compliant data from all regions into a single analytics platform for global insights.
- **Local-First Operations**: Maintain full operational capability in each region even during network outages or compliance restrictions.
- **Flexible Routing**: Send different data types to different destinations - raw events stay regional, aggregated metrics go global.

## Benefits of Distributed Edge Analytics

**Compliance & Governance**

- Meet GDPR, CCPA, and data sovereignty requirements through regional processing
- Automatic PII redaction and anonymization before data leaves regions
- Maintain audit trails showing where and how data was processed
- Ensure data residency requirements without sacrificing analytics value

**Performance & Scale**

- Reduce central analytics load by pre-aggregating data at regional edges
- Faster queries on smaller, pre-processed datasets vs. raw global data
- Scale processing horizontally across regions rather than centralizing
- Maintain regional responsiveness with local processing

**Operational Resilience**

- Regional operations continue during network partitions
- Local analytics and dashboards remain available regardless of central connectivity
- Automatic backfill when connectivity to central systems is restored
- No single point of failure in analytics infrastructure

## Common Patterns

**Regional Aggregation with Global Rollup**
Process and aggregate data within each region (EU, US, APAC), sending only aggregated metrics and anonymized datasets to global analytics platforms. Raw data never crosses regional boundaries.

**Tiered Data Sovereignty**
Keep personally identifiable data and sensitive business information within regions while sharing anonymized trends, aggregates, and insights globally.

**Multi-Region Compliance**
Apply region-specific privacy rules automatically: GDPR redaction in EU, CCPA compliance in California, industry regulations in specific countries - all from centralized pipeline management.

**Federated Querying**
Run analytics queries that combine results from multiple regional processing systems, each enforcing local compliance rules before contributing to the unified result.

**Lineage-Ready Event Streams**
The [`metadata` processor](/components/processors/metadata) produces [OpenLineage](https://openlineage.io/)-aligned identity fields (`run_id`, `job_name`, `job_namespace`, `event_time`, `producer`) on every message, so events flowing into your global analytics platform are already shaped for lineage tooling. Pair this with the edge's native [OpenLineage emitter](/guides/pipelines/lineage), which fires a signed lifecycle event at every pipeline transition — the `run_id` on each message matches the `runId` on those events, so downstream catalogs can correlate per-record context back to per-run lineage. For the full audit and governance story, see the [Data Lineage use case](/use-cases/data-lineage).

```yaml
pipeline:
  processors:
    - metadata:
        include: [core, pipeline]
        target: body
        format: nested
        body_key: lineage
```

**Local-First Dashboards**
Serve operational dashboards from regional edge systems for immediate insights, while background processes sync aggregated data to central business intelligence platforms.

## Example Use Cases

- **Global retail chains** processing transaction data regionally to comply with local privacy laws while maintaining centralized sales analytics and inventory optimization
- **Healthcare networks** analyzing patient data within regional hospital systems, sharing only aggregated clinical insights for research while maintaining HIPAA compliance
- **Financial services** performing fraud detection regionally with local models, aggregating threat intelligence globally while keeping customer data in-region
- **Multi-national SaaS platforms** processing user analytics regionally per GDPR/CCPA requirements while providing product teams with global usage insights

## Production Examples

<div className="grid md:grid-cols-2 gap-4 my-6">
  
  
  
  
</div>

## Next Steps

- **[Quick Start Guide](/getting-started/quickstart/)**: Build your first distributed analytics pipeline
- **[Bloblang Transformations](/guides/bloblang)**: Learn data transformation and anonymization techniques
- **[Branch Processor](/components/processors/branch)**: Route data to different regional destinations
- **[Mapping Processor](/components/processors/mapping)**: Apply regional compliance rules
- **[Kafka Output](/components/outputs/kafka)**: Aggregate regional data to central systems

## IoT Data Aggregation

# IoT Data Aggregation

## Overview

IoT deployments generate massive volumes of high-frequency data from sensors, devices, and equipment. Sending every reading to the cloud creates extreme bandwidth costs, storage challenges, and delayed anomaly detection. Processing data at the edge - aggregating, downsampling, and analyzing locally before transmission - dramatically reduces costs while improving real-time responsiveness.

## Expanso's Approach to IoT Data Aggregation

Expanso Edge processes sensor data at the source, running on edge gateways or industrial PCs near your IoT devices. Agents receive high-frequency readings via MQTT or HTTP, perform aggregation and analysis locally, and send only meaningful insights to cloud systems.

**Key capabilities:**

- **Time-Window Aggregation**: Collect high-frequency readings (per-second) and aggregate into configurable intervals (per-minute, per-hour), calculating statistics like mean, min, max, and percentiles.
- **Anomaly Detection**: Identify outliers, sudden changes, and pattern deviations in real-time at the edge, triggering immediate alerts before data reaches the cloud.
- **Downsampling with Fidelity**: Send fewer records through statistical summaries and edge-based filtering. Choose the statistics needed by downstream consumers; summaries cannot preserve every raw reading.
- **Protocol Flexibility**: Ingest data from MQTT brokers or HTTP endpoints, all in a single pipeline.
- **Offline Resilience**: Configure buffering and retry behavior for outages. Retention depends on buffer capacity, persistence, source acknowledgements, and outage duration.

## Benefits of Edge IoT Processing

Processing IoT data at the edge provides significant advantages:

**Cost Reduction**

- Reduce bandwidth use when summaries are smaller than the raw readings they replace
- Lower cloud ingestion costs by sending summaries instead of raw readings
- Minimize storage costs with pre-filtered, aggregated datasets
- Measure ingest, storage, edge compute, and network costs together to estimate savings for your workload

**Real-Time Operations**

- Sub-second anomaly detection and alerting at the source
- Local dashboards provide instant visibility without cloud latency
- Immediate automated responses to critical sensor readings
- Maintain operations during network outages with local processing

**Data Quality**

- Preserve analytical fidelity through statistical aggregation (mean, percentiles, variance)
- Use appropriately sized durable buffers to reduce loss during downstream outages
- Monitor gaps and buffer exhaustion during network interruptions
- Keep processing in selected regions as one part of your data sovereignty controls

## Worked aggregation example

A sensor reporting once per second produces 60 records per minute. Replacing those records with one minute-level summary reduces the record count by 59 / 60, or about 98.3%. This is an illustrative record-count calculation, not a measured bandwidth or cost result: summary size, metadata, compression, retries, and edge compute affect the actual savings. Keep raw readings when investigations or downstream calculations require them.

## Common Patterns

**Statistical Aggregation**
Collect per-second sensor readings and aggregate into minute or hour intervals. Calculate mean, min, max, standard deviation, and percentiles according to the analytical requirements of downstream consumers.

**Threshold-Based Alerting**
Monitor sensor readings against configurable thresholds at the edge. Trigger immediate alerts when values exceed limits, fall outside normal ranges, or show sudden rate-of-change anomalies.

**Multi-Protocol Collection**
Combine data from MQTT sensors and HTTP devices in a single pipeline. Normalize formats and enrich with location metadata before aggregation.

**Tiered Storage**
Send aggregated summaries to cloud analytics platforms every few minutes, while keeping high-frequency raw data locally for short-term analysis and backfill operations.

**Edge Dashboards**
Serve real-time operational dashboards directly from edge locations using local Grafana or custom HTTP endpoints, eliminating cloud round-trips for monitoring.

## Example Use Cases

- **Manufacturing plants** aggregating sensor data from thousands of machines, detecting equipment anomalies locally, and sending only summaries to centralized analytics
- **Smart buildings** processing HVAC, occupancy, and energy sensors in real-time, optimizing operations locally while reporting trends to cloud systems
- **Agriculture deployments** monitoring soil moisture, temperature, and weather data across remote farms with intermittent connectivity
- **Energy infrastructure** collecting high-frequency telemetry from solar arrays, wind turbines, and grid equipment, detecting faults immediately while archiving aggregated performance data

## Production Examples

<div className="grid md:grid-cols-2 gap-4 my-6">
  
  
  
  
  
</div>

## Next Steps

- **[Quick Start Guide](/getting-started/quickstart/)**: Build your first IoT aggregation pipeline
- **[Bloblang Transformations](/guides/bloblang)**: Learn data transformation and aggregation techniques
- **[MQTT Input](/components/inputs/mqtt)**: Receive data from IoT sensors
- **[Mapping Processor](/components/processors/mapping)**: Aggregate and transform sensor readings
- **[HTTP Server Output](/components/outputs/http_server)**: Serve local dashboards

## Log Processing

# Log Processing at Edge

## Overview

Modern distributed systems generate massive volumes of logs across edge locations, data centers, and cloud environments. Traditional centralized logging approaches struggle with high data transfer costs, slow query performance, and compliance challenges. Processing logs at the edge - where they originate - addresses these fundamental challenges while maintaining complete visibility into system health.

## Expanso's Approach to Log Processing

Expanso Edge transforms log management by processing data at the source before sending it to centralized systems. Edge agents run directly on your infrastructure, applying intelligent filtering, aggregation, and redaction in real-time.

**Key capabilities:**

- **Intelligent Filtering**: Remove debug-level noise and duplicate events at the source, typically reducing volume by 70-85% before transmission.
- **PII Redaction**: Automatically detect and remove sensitive data (credit cards, SSNs, API keys) before logs leave the edge.
- **Pre-Aggregation**: Calculate metrics and statistics locally, sending summaries instead of raw events.
- **Multi-Destination Routing**: Send errors to search indexes, metrics to monitoring systems, and archives to cold storage - all from a single pipeline.
- **Real-Time Alerting**: Detect critical patterns instantly at the edge, triggering alerts before logs reach centralized systems.

## Benefits of Edge Log Processing

Processing logs at the edge provides significant operational and cost advantages:

**Cost Reduction**

- Reduce bandwidth usage by 70-90% through intelligent filtering and compression
- Lower cloud ingestion costs by sending only relevant events
- Minimize storage costs by archiving pre-filtered data
- Typical savings: 80%+ reduction in total logging costs

**Performance & Compliance**

- 10x faster queries by searching smaller, filtered datasets
- Sub-second alerting on critical events vs. minutes with centralized processing
- Automatic compliance with data protection regulations through source-level PII redaction
- Regional data processing meets data sovereignty requirements

**Operational Excellence**

- Centralized pipeline management across thousands of edge locations
- Consistent processing rules deployed from a single interface
- Real-time visibility into log volumes and processing metrics
- Simplified troubleshooting with location-aware enrichment

## Common Patterns

**Severity-Based Filtering**
Keep only warnings, errors, and critical logs while discarding verbose debug output. This single filter typically provides 70-85% volume reduction.

**Sensitive Data Protection**
Apply regex-based redaction to remove PII, credentials, and other sensitive information before transmission. Ensures compliance with GDPR, HIPAA, and other data protection regulations.

**Metric Extraction**
Calculate error rates, latency percentiles, and custom business metrics at the edge. Send aggregated metrics to monitoring systems while archiving detailed events to cold storage.

**Conditional Routing**
Route different log types to appropriate destinations: errors to Elasticsearch for analysis, metrics to Prometheus, critical alerts to Slack/PagerDuty, and all logs to S3 for archival.

**Context Enrichment**
Add location, environment, and deployment metadata at the edge. Enables filtering and grouping in centralized systems without complex correlation queries.

## Example Use Cases

- **Retail chains** processing point-of-sale logs across hundreds of stores, filtering transaction noise while capturing all errors and compliance events
- **Manufacturing facilities** monitoring equipment logs, aggregating metrics locally, and alerting on anomalies before sending summaries to centralized analytics
- **Financial services** redacting PII from application logs, ensuring compliance while maintaining audit trails
- **SaaS platforms** filtering high-volume debug logs from edge regions, routing errors for analysis while archiving filtered events

## Production Examples

<div className="grid md:grid-cols-2 gap-4 my-6">
  
  
  
  
  
</div>

## Next Steps

- **[Quick Start Guide](/getting-started/quickstart/)**: Build your first edge log processing pipeline
- **[Bloblang Transformations](/guides/bloblang)**: Learn the transformation language for filtering and enrichment
- **[File Input](/components/inputs/file)**: Read logs from files or stdout
- **[Mapping Processor](/components/processors/mapping)**: Transform and filter log events
- **[S3 Output](/components/outputs/aws_s3)**: Archive processed logs

## Real-Time Alerting

# Real-Time Alerting

## Overview

Detecting critical events quickly is essential for operational excellence, but traditional centralized monitoring introduces latency - data must traverse the network, be ingested, indexed, and queried before alerts fire. This delay can mean minutes between an incident occurring and teams being notified. Processing data at the edge enables sub-second detection and alerting, triggering notifications before data even reaches centralized systems.

## Expanso's Approach to Real-Time Alerting

Expanso Edge runs detection logic directly at data sources, analyzing events in real-time and triggering alerts immediately upon pattern detection. Edge agents apply rules, thresholds, and pattern matching locally, sending alerts through multiple channels while continuing to forward data for historical analysis.

**Key capabilities:**

- **Sub-Second Detection**: Analyze events as they occur at the edge, triggering alerts in milliseconds rather than minutes.
- **Pattern-Based Detection**: Identify complex patterns across event sequences, not just simple threshold breaches - detect emerging issues before they become critical.
- **Multi-Channel Alerting**: Send alerts to Slack, PagerDuty, email, webhooks, or custom endpoints directly from the edge without central coordination.
- **Intelligent Filtering**: Apply de-duplication, rate limiting, and severity-based routing at the edge to reduce alert fatigue and noise.
- **Contextual Enrichment**: Automatically add location, environment, and historical context to alerts at the source, improving incident response.

## Benefits of Edge Alerting

**Faster Incident Response**

- Reduce mean time to detection (MTTD) from minutes to sub-second
- Alert on critical issues before data reaches centralized monitoring
- Enable automated responses to events at the source
- Maintain alerting capability during network outages

**Reduced Alert Fatigue**

- Filter and de-duplicate alerts at the edge before generating notifications
- Apply intelligent rate limiting to prevent storm conditions
- Route different severity levels to appropriate channels automatically
- Suppress known issues during maintenance windows without central coordination

**Improved Context**

- Enrich alerts with edge-specific metadata: location, equipment ID, recent trends
- Include relevant event history in alert payloads for faster diagnosis
- Correlate events across local systems before creating alerts
- Maintain full event context even when centralized systems are unavailable

## Common Patterns

**Threshold-Based Alerting**
Monitor metrics against static or dynamic thresholds, triggering alerts when values exceed limits. Apply at the edge for immediate notification rather than waiting for centralized metric aggregation.

**Error Pattern Detection**
Identify sequences of errors, sudden spikes in failure rates, or specific error patterns that indicate systemic issues. Alert immediately while buffering events for later analysis.

**Anomaly Detection**
Apply statistical analysis with Bloblang, or run models locally with the `ollama` processors or hosted models with the `openai` processors, to flag unusual behavior such as sudden metric changes, unexpected patterns, or deviations from historical baselines.

**Multi-Event Correlation**
Combine events from multiple sources at the edge to detect complex scenarios: multiple related failures, cascading issues, or coordinated attack patterns.

**Rate-Based Alerting**
Trigger alerts when event rates exceed thresholds: too many errors per minute, unusual request volumes, or unexpected traffic patterns - detected immediately at the source.

## Example Use Cases

- **E-commerce platforms** detecting payment processing failures at the edge, alerting teams instantly while orders are still in flight, enabling real-time remediation
- **Manufacturing facilities** monitoring equipment sensor readings at the source, triggering immediate shutdown procedures for dangerous conditions before data reaches cloud systems
- **Security operations** analyzing authentication logs at the edge, detecting brute force attacks or credential stuffing in real-time and triggering automatic IP blocking
- **SaaS platforms** identifying service degradation at regional edge points, alerting operations teams and triggering circuit breakers before issues impact broader user base

## Production Examples

<div className="grid md:grid-cols-2 gap-4 my-6">
  
  
  
  
</div>

## Next Steps

- **[Quick Start Guide](/getting-started/quickstart/)**: Build your first edge alerting pipeline
- **[Bloblang Transformations](/guides/bloblang)**: Learn pattern detection and filtering techniques
- **[Mapping Processor](/components/processors/mapping)**: Apply detection rules and thresholds
- **[Branch Processor](/components/processors/branch)**: Route alerts to different channels by severity
- **[HTTP Client Output](/components/outputs/http_client)**: Send alerts to webhooks and APIs

## Security & Compliance

# Security & Compliance

## Overview

Organizations face increasing regulatory requirements around data privacy, security, and auditability. PII exposure, data breaches, and compliance violations often occur during data collection and transmission - before security controls are applied. Processing data at the edge enables enforcement of security policies and compliance rules at the source, preventing violations before sensitive data enters your infrastructure.

## Expanso's Approach to Security & Compliance

Expanso Edge applies security policies and compliance rules at the point of data collection, before transmission or storage. Edge agents automatically detect, redact, and control sensitive data according to configurable policies managed centrally.

**Key capabilities:**

- **Automated PII Detection**: Identify and redact credit cards, SSNs, email addresses, phone numbers, and other PII using regex patterns, or model-based detection via the `ollama_chat` and `openai_chat_completion` processors.
- **Policy Enforcement at Source**: Apply data minimization, anonymization, and encryption policies before data leaves edge locations.
- **Audit Trail Generation**: Automatically create tamper-evident audit logs showing what data was collected, transformed, and transmitted - required for GDPR, HIPAA, and SOC 2 compliance.
- **Encryption in Motion**: Apply field-level encryption or format-preserving encryption at the edge, ensuring sensitive data is protected throughout its lifecycle.
- **Consent Management**: Filter and route data based on user consent preferences, ensuring only authorized data flows to specific destinations.

## Benefits of Edge Security & Compliance

**Risk Reduction**

- Eliminate PII from datasets before transmission, reducing breach exposure surface
- Prevent accidental logging or storage of sensitive credentials and tokens
- Apply defense-in-depth by enforcing policies at collection, not just at storage
- Reduce blast radius of security incidents through source-level data minimization

**Compliance Automation**

- Meet GDPR "privacy by design" requirements through automated edge redaction
- Ensure HIPAA compliance by de-identifying PHI before transmission
- Satisfy PCI-DSS requirements by preventing credit card data from entering logs or analytics
- Generate audit trails required for SOC 2, ISO 27001, and other certifications

**Operational Efficiency**

- Centrally manage compliance policies deployed to thousands of edge locations
- Automatic application of new regulations without code changes
- Reduce manual audit workload through automated compliance logging
- Simplify data retention policies by collecting only necessary, compliant data

## Common Patterns

**Automated PII Redaction**
Scan all data for patterns matching credit cards, SSNs, email addresses, phone numbers, and custom sensitive patterns. Replace detected values with tokens or redacted placeholders before data leaves the edge.

**Field-Level Encryption**
Encrypt specific fields containing sensitive data at the edge using customer-managed keys. Only authorized downstream systems with decryption keys can access the original values.

**Consent-Based Routing**
Check user consent preferences and route data accordingly: analytics-approved data to business intelligence, essential-only data to operational systems, full opt-out data is filtered entirely.

**Audit Log Enrichment**
Automatically append compliance metadata to all events: who collected the data, when, under what policy, what transformations were applied, and where it was sent. The [`metadata` processor](/components/processors/metadata) makes this declarative — attach the node's `region`, `environment`, and a custom `compliance_tier` to every message without writing Bloblang:

```yaml
pipeline:
  processors:
    - metadata:
        include: [node]
        custom:
          compliance_tier: pii
          retention_policy: gdpr-7-year
        target: body
        format: nested
        body_key: compliance
```

**Data Minimization**
Apply principle of least privilege to data collection: filter unnecessary fields, aggregate granular data, and retain only what's required for legitimate business purposes.

## Example Use Cases

- **Healthcare providers** de-identifying patient records at the source, maintaining HIPAA compliance while enabling clinical analytics and research
- **Retail companies** redacting payment card data from logs and analytics streams, preventing PCI-DSS scope expansion while maintaining fraud detection capability
- **SaaS platforms** enforcing user privacy preferences at the edge, routing data according to consent levels before it enters analytics pipelines
- **Financial institutions** detecting and tokenizing account numbers, SSNs, and sensitive customer data in real-time logs before centralized storage

## Production Examples

<div className="grid md:grid-cols-2 gap-4 my-6">
  
  
  
  
  
</div>

## Next Steps

- **[Quick Start Guide](/getting-started/quickstart/)**: Build your first compliance-focused pipeline
- **[Bloblang Transformations](/guides/bloblang)**: Learn PII detection and redaction techniques
- **[Mapping Processor](/components/processors/mapping)**: Apply redaction and transformation rules
- **[Branch Processor](/components/processors/branch)**: Route data based on compliance policies
- **[Kafka Output](/components/outputs/kafka)**: Send compliant data to downstream systems


## Full Reference

See https://docs.expanso.io/use-cases for complete documentation.