Skip to main content

CI Test Fixtures

Generate a synthetic dataset for tests in which every field is derived from the record number, with no random values and no clock, so every run writes the same file, byte for byte. Use expanso-edge validate in the same pipeline to fail the build when a job config has a typo.

Components: generate input (count: 200) → file output, with pipeline.threads: 1.

Needs: nothing beyond a node. The job reads no external data.

Proved on a local-mode node

This job ran end to end on an expanso-edge v2.1.21 node in local mode, not through Expanso Cloud.

Complete job​

ci-fixtures.yaml
name: ci-fixtures
type: pipeline
# One-shot job: stop after one failed execution instead of retrying.
restart_policy: never
selector:
  match_labels:
    pipeline_role: ci
config:
  input:
    generate:
      count: 200
      interval: ""
      mapping: |
        let n = count("fixtures_RUN_ID")
        let first = [
          "Ada", "Grace", "Alan", "Edsger", "Barbara", "Ken",
          "Frances", "Donald"
        ]
        root.id = "cust-%05d".format($n)
        root.name = "%s Tester%d".format($first.index($n % 8), $n)
        root.email = "tester%[email protected]".format($n)
        root.tier = ["gold", "silver", "bronze"].index($n % 3)
        root.balance = "%d.%02d".format(
          ($n * 7919 % 100000 / 100).floor().int64(),
          ($n * 7919 % 100).int64()
        )
        root.created_at = (1767225600 + $n * 86400).ts_format(
          "2006-01-02T15:04:05Z", "UTC"
        )
        root.active = $n % 7 != 0
  pipeline:
    # One processing thread keeps output order equal to generation
    # order. With the default parallelism the same records came out
    # in a different order on each run, so the file was not
    # byte-identical.
    threads: 1
    processors: []
  output:
    file:
      path: "/var/tmp/expanso-fixtures/fixtures.jsonl"
      codec: lines

Two settings make the output repeatable:

  • threads: 1. With the default parallelism, the same 200 records came out in a different order on each run. One thread keeps them in generation order.
  • A new counter name for every run. A count() counter keeps counting for as long as the node process runs, across jobs. Replace RUN_ID with a value that is unique to the run, such as your CI build number, so the ids start at cust-00001 every time.

The selector sends the job to a node labelled pipeline_role: ci; set that label on your node, or change the selector to a label it already carries. With no matching node, the job is stored and never runs.

Use it in CI​

Check the config first: expanso-edge validate exits with a non-zero status on an unknown component or field, which fails the step. Then substitute the run id and deploy:

expanso-edge validate ci-fixtures.yaml
sed "s/RUN_ID/$BUILD_ID/" ci-fixtures.yaml > ci-fixtures-run.yaml
expanso-cli job deploy ci-fixtures-run.yaml

BUILD_ID stands for your CI system's unique build number. The output path is on the node that runs the job; create its directory there first with mkdir -p /var/tmp/expanso-fixtures. Each run in the proof wrote to a fresh file, so remove the previous fixtures.jsonl before the next run, and remove the finished job with expanso-cli job delete ci-fixtures --yes (without --yes the command waits at a confirmation prompt, which fails in a non-interactive CI step).

A misspelled component name fails like this, from the run:

(line 10, col 3) [input]. Unknown component or field 'generat'. Check that the component name is spelled correctly.

Check it​

Every record matches this JSON Schema. The run checked all 200 against it:

schema.json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "id",
    "name",
    "email",
    "tier",
    "balance",
    "created_at",
    "active"
  ],
  "properties": {
    "id": {
      "type": "string",
      "pattern": "^cust-[0-9]{5}$"
    },
    "name": {
      "type": "string",
      "minLength": 3
    },
    "email": {
      "type": "string",
      "pattern": "^tester[0-9]+@example\\.invalid$"
    },
    "tier": {
      "enum": [
        "gold",
        "silver",
        "bronze"
      ]
    },
    "balance": {
      "type": "string",
      "pattern": "^[0-9]+\\.[0-9]{2}$"
    },
    "created_at": {
      "type": "string",
      "pattern": "^20[0-9]{2}-[0-9]{2}-[0-9]{2}T00:00:00Z$"
    },
    "active": {
      "type": "boolean"
    }
  }
}

The first line of the file, pretty-printed:

{
  "active": true,
  "balance": "79.19",
  "created_at": "2026-01-02T00:00:00Z",
  "email": "[email protected]",
  "id": "cust-00001",
  "name": "Grace Tester1",
  "tier": "silver"
}

What the run proved​

  • Two runs each wrote 200 records, cust-00001 to cust-00200 in order.
  • The two files were byte-identical: the same SHA-256.
  • Every record validated against the schema above with an independent JSON Schema checker.
  • A copy of the job with generate misspelled as generat made expanso-edge validate exit 1.
  • The job exactly as published, restart_policy: never included, ran on a local-mode node running expanso-edge v2.1.21. Each of the two runs completed after 1 execution.

Limits​

  • Proved on a local-mode node only, not through Expanso Cloud.
  • With restart_policy: never, a failed run stops after one execution instead of being re-run. See Proving your own run.
  • The dataset is fixed by the mapping. Edit the mapping to change the fields; keep every value derived from the counter to stay deterministic.
  • The output is a file on the node. To use it in a CI job on another machine, copy it from the node or replace the file output.