Skip to main content

RSS Feed Engine

Poll an RSS 2.0 feed, turn each item into a JSON record, and keep a file-backed record of every item already seen, so no item is emitted twice, even after the job restarts.

Components: generate (the poll schedule) → http → mapping (fetch error check) → xml (to_json) → mapping → unarchive → mapping → dedupe with a file cache → file output.

Needs: a node with outbound HTTPS access to the feed. No credentials.

Complete job​

The job below polls the NASA news-release feed once. The selector sends it to a node labelled pipeline_role: feeds; set that label on your node, or change the selector to a label it already carries. With no matching node, the job is stored and never runs.

The feed URL appears twice in the job: in http.url and in the mapping's root.feed. To switch feeds, change both together.

rss-feed-engine.yaml
name: rss-feed-engine
type: pipeline
# One-shot job: stop after one failed execution instead of retrying.
restart_policy: never
selector:
  match_labels:
    pipeline_role: feeds
config:
  input:
    generate:
      # One bounded poll. For a standing engine, drop `count` and set
      # `interval` to the feed's polling period, e.g. "5m". Also
      # remove `restart_policy: never` above: one transient fetch
      # error would stop a long-running job for good. The default
      # policy restarts it on failure.
      count: 1
      interval: 1s
      mapping: root = {}
  pipeline:
    processors:
      - http:
          url: "https://www.nasa.gov/news-release/feed/"
          verb: GET
          retries: 3
      # Stop here with the fetch error before XML parsing.
      - mapping: |
          root = if errored() {
            throw("feed fetch failed: " + error().or("unknown"))
          } else { content() }
      - xml:
          operator: to_json
      - mapping: |
          let items = this.rss.channel.item
          root = if $items.type() == "array" { $items } else {
            [$items]
          }
      - unarchive:
          format: json_array
      - mapping: |
          let g = this.guid
          root.guid = if $g.type() == "object" { $g."#text" } else {
            $g
          }
          root.title = this.title
          root.link = this.link
          root.category = this.category
          root.summary = this.description
          # RSS 2.0 dates are RFC 822: numeric zone (+0000) or a
          # name (GMT). Bind to a variable: inside .catch(...)
          # `this` is not the item.
          let pd = this.pubDate
          root.published_at = $pd.ts_strptime(
            "%a, %d %b %Y %H:%M:%S %z"
          ).catch($pd.ts_strptime(
            "%a, %d %b %Y %H:%M:%S %Z"
          )).ts_format("2006-01-02T15:04:05Z07:00", "UTC")
          root.feed = "https://www.nasa.gov/news-release/feed/"
      - dedupe:
          cache: seen_guids
          key: '${! json("guid").hash("sha256").encode("hex") }'
  cache_resources:
    - label: seen_guids
      file:
        directory: "/var/tmp/expanso-rss/state/seen-guids"
  output:
    file:
      path: "/var/tmp/expanso-rss/out/rss-items.jsonl"
      codec: lines

The two paths are on the node that runs the job, not the machine you deploy from, and the user running the agent must be able to write to them. Create the directories on that node first:

mkdir -p /var/tmp/expanso-rss/state/seen-guids \
  /var/tmp/expanso-rss/out

Deploy and check it​

expanso-cli job validate rss-feed-engine.yaml --offline
expanso-cli job deploy rss-feed-engine.yaml
expanso-cli job describe rss-feed-engine

Then, on the node, count what arrived and compare it with the feed:

wc -l < /var/tmp/expanso-rss/out/rss-items.jsonl
curl -sL https://www.nasa.gov/news-release/feed/ \
  | grep -o '<item>' | wc -l

A live feed changes over time, so compare the two counts close together, and expect a difference if the feed published or dropped items in between. Run the job again with the same cache directory and the file gains lines only for items it has not seen before.

Each line is one item with the fields guid, title, link, category, summary, published_at and feed. One record from the run on 2026-09-18 (UTC), with its long link, category and summary values left out; today's feed will carry different items:

{
  "guid": "https://www.nasa.gov/?post_type=press-release&p=1048343",
  "title":
    "NASA Awards Launch Services for StarBurst Gamma-Ray Detector",
  "published_at": "2026-09-17T20:37:09Z",
  "feed": "https://www.nasa.gov/news-release/feed/"
}

What the run proved​

  • Live feeds, 2026-09-18. One poll of the NASA feed produced 10 records and one poll of the BBC Technology feed produced 21. In both cases the GUIDs matched an independent fetch of the same feed taken before and after the run. BBC dates use GMT and NASA dates use +0000; the date mapping handles both.
  • Deduplication across restarts. Against a local test feed containing a duplicate GUID, the job emitted 4 distinct items. A second run with the same cache directory emitted 0.
  • Where it ran. Each job above was deployed through Expanso Cloud to a labelled node running expanso-edge v2.1.21. Those runs used an earlier job without the fetch error check and without restart_policy.
  • The published job, local-mode node only. The job exactly as published, fetch error check and restart_policy: never included, was run on a local-mode node running v2.1.21. Against a local test feed it completed after 1 execution with 4 distinct items, and a second run with the same cache emitted 0. A 401 and a 503 response each failed the job after exactly 1 execution with 0 records, and the error named the fetch. A live poll of the NASA feed completed after 1 execution, and its GUIDs equalled an independent fetch. Without the check, a 401 also failed the job with 0 records, at the xml step instead: the check makes the failure name the fetch, it does not change whether the job fails.

Limits​

  • The runs used bounded polls (count: 1 against the live feeds, count: 3 against the test feed). A standing engine drops count and sets interval, as the comment in the job says; that schedule was not part of the runs. Such a long-running job should not keep restart_policy: never, or a single transient fetch error stops it for good. Choose a policy suited to a long-running job; the default restarts it on failure.
  • The output is a file on the node. To send items elsewhere, replace the file output with another output, such as http_client.
  • The mapping reads RSS 2.0 (rss.channel.item). An Atom feed uses a different structure (feed.entry) and needs its own mapping.
  • The dedupe key is the item's guid, which RSS 2.0 makes optional. Items without one are never emitted: hashing the missing guid fails, so every such item is dropped and the run is marked failed. If your feed omits guid, key on another field such as link instead; that variant was not run.
  • The published job, with restart_policy: never, was proved on a local-mode node only; the Cloud runs used the earlier job without it. With never, a failed run stops after one execution instead of being re-run. See Proving your own run.