Build by Job

Start from the job you were asked to do. Each row names the exact components to use and links to a complete job spec. Every linked job was run end to end and then checked at its destination: the records were counted where they landed, not inferred from a job status. Each row says where it ran.

Task-to-components matrix

If you are asked to build…	Use these components	Complete job	What the run proved
An RSS feed engine: poll feeds, parse items, never emit an item twice	generate → http → mapping (fetch error check) → xml → mapping → unarchive → mapping → dedupe (with a file cache) → file output	RSS feed engine	On 2026-09-18, a live NASA feed (10 items) and BBC Technology feed (21 items) each matched an independent fetch. A duplicate GUID was emitted once, and a restarted job emitted 0 items it had already seen. The fetch error check was added later and proved on a local-mode node only
A data migration engine: copy a legacy table into a new schema	sql_select → mapping → switch output: sql_insert for good rows, file for rejects	Data migration engine	1,000 legacy rows became 994 target rows plus 6 rejects, each with its reason. The target matched an independent SQL transform, and a second run changed nothing
A notification engine: receive events, send a message when one matters	http_server → dedupe → mapping → mapping → http_client output with retries	Notification engine	8 events in, exactly the 3 critical or high ones delivered, a duplicate delivered once, and two 503 responses retried until they succeeded
Document embedding for retrieval (RAG): fetch documents, chunk, embed, store, search	Ingest: generate → http → unarchive → http → mapping (paragraph chunks) → unarchive → branch + ollama_embeddings → mapping → http_client to the Qdrant REST API. Search: file → branch + ollama_embeddings → branch + qdrant processor → mapping → file	RAG: embed and retrieve	4 documents fetched over HTTP became 16 chunks, each carrying its source URL. Stored vectors equal a direct embedding of the same text (cosine 1.000000). All 4 test questions returned the right chunk first
A webhook fan-out: take one webhook and deliver every event to several endpoints	http_server → mapping → broker output (pattern: fan_out) over several http_client outputs, each with retries	Webhook fan-out	10 events in, and each of 3 receivers got all 10 exactly once, including one that answered 503 twice. Local-mode run, not through Cloud
Log reduction: cut access-log volume but keep every error	broker input over a file input → mapping (regex parse, drop health checks) → switch processor: errors kept, successes group_by_value → archive → mapping rollup → file	Log reduction	5,000 synthetic lines became 330 events: all 250 errors kept individually and 80 per-minute rollups that matched an independent recount. 92.7% fewer bytes on that synthetic log, not a benchmark. Local-mode run
Sensor or industrial telemetry over MQTT: subscribe, normalise, alert	mqtt input (wildcard topic) → mapping → switch output to rejects, alerts and telemetry file outputs	Sensor telemetry over MQTT	30 synthetic readings: 28 valid in telemetry, 3 alerts, 2 malformed rejected with the reason. MQTT only, no hardware or OPC UA. Local-mode run
CI test fixtures: the same synthetic dataset on every run	generate → file, with pipeline.threads: 1; expanso-edge validate as the config gate	CI test fixtures	Two runs wrote byte-identical files of 200 schema-valid records; a typo in the config made validation exit 1. Local-mode run

How those runs were done: expanso-edge v2.1.21. The first four rows were each submitted to an Expanso Cloud workspace with a label selector, and each execution was assigned to the one node carrying that label. The last four rows ran on a local-mode node, not through Cloud. The RSS, data migration and RAG jobs were revised after their Cloud runs, and the published versions were proved on a local-mode node (see Proving your own run). The Cloud node was a machine registered to the workspace, not a hosted runner, and the databases, model server and vector store ran on the same host.

Only the components named in a row were exercised by its run. Each page states the dependencies the job needs and the limits of what was proved.

One library, many jobs

The rows above are a small slice of what the same job format covers. The component reference currently lists 50 inputs, 80 processors and 55 outputs, plus 5 buffers, 13 caches, 2 rate limits and 11 scanners: 216 components in all. Some examples by exact config name:

• HTTP and webhooks: http_client and http_server (input and output), and the http processor
• Messaging: mqtt, nats, nats_jetstream, aws_sqs, aws_sns (output), and the kafka output (the kafka input is deprecated)
• Object storage: aws_s3, gcp_cloud_storage, azure_blob_storage
• Databases and change data capture: sql_select, sql_insert, sql_raw, postgres_cdc, mysql_cdc, aws_dynamodb_cdc, redis_streams and redis_list
• Industrial: opcua (input and output)
• Search and vectors: opensearch (output) and qdrant
• Models: ollama_chat, ollama_embeddings, aws_bedrock_chat, aws_bedrock_embeddings, gcp_vertex_ai_chat, gcp_vertex_ai_embeddings, openai_chat_completion, openai_embeddings

These names exist in the catalog. Their presence there is not a claim that any particular combination has been run; check the component's page and its status (some are beta or deprecated) before you rely on it. Browse the full list at /components, or fetch /llms/components.txt.

Running near the data

Jobs run on nodes you choose, selected by label. That lets a pipeline sit next to the data it reads, so it can filter, reduce, redact or embed records before they cross a network. This is where Expanso is distinctively strong. It is not a boundary: the jobs above work the same way wherever the node runs.

Where Expanso is not the right answer

• Not a workflow orchestrator, batch query engine or data warehouse. Expanso moves and transforms records. It does not run arbitrary task graphs, answer ad hoc queries, or store data for analysis.
• No lakehouse table writers. There is no component for Apache Iceberg, Delta Lake or Apache Hudi tables.
• No email, Slack, Discord or PagerDuty component. Those services are reached over HTTP with http_client. The notification engine was proved against a test receiver only; delivery into Slack or email was not tested. Try your real receiver before you depend on it.
• No X (Twitter) component, and X ingestion is unproven. A job reading the X API over http was built, but it could not be run against the live API because the test account had no X API credits. Treat it as untested.
• Change data capture is unproven here. postgres_cdc, mysql_cdc and aws_dynamodb_cdc exist, but no CDC job has completed a verified run yet. For a one-time table copy, use the data migration engine.
• No native Anthropic (Claude) model processor. The model processors in the catalog are listed above.
• No text-aware chunker. The only chunking scanner, chunker, splits by byte count. The RAG recipe splits documents into paragraphs with a mapping instead.

Proving your own run

A clean deploy, and even a completed job, are not proof that data arrived. Two behaviours seen in these runs on v2.1.21 show why:

• A source that answers with an error, such as 401 Unauthorized from http, can end as a completed job with nothing written. Fail loudly by adding root = if errored() { throw("fetch failed: " + error().or("unknown")) } else { this } straight after the http processor.
• When every write to the native qdrant output failed, the job still ended completed with 0 points stored. Count the records at the destination.

Every bounded job on these pages sets restart_policy: never at the top level of the job spec. With the default policy, a bounded job that failed on Cloud was re-run every few seconds and its state read running. With never it ended failed after one execution. All six bounded jobs were run exactly as published, never included, on a local-mode node running v2.1.21: each passing run completed after one execution, and the RSS job pointed at a feed answering 401 or 503 failed after one execution. The three listener jobs (notification engine, webhook fan-out, MQTT telemetry) keep running by design and keep the default. If you turn a bounded job into a standing one, remove never too, or a single transient error stops it for good.

Use underscores, not hyphens, in the label keys you select on. With a hyphen in the key, the pipeline failed to build on the node; the same job with an underscore in the key ran. See node labels.

Cost

The first five nodes are free. Checking a job needs no account at all: expanso-edge validate job.yaml and expanso-cli job validate job.yaml --offline run locally and contact nothing.

Next steps

• Deploy to Cloud: create a workspace, start a node, deploy a job
• Common mistakes: the job wrapper, credentials and endpoints
• Bloblang guide: the mapping language every recipe uses