Pipeline Progress Lost After Restart

When the edge process restarts (crash, kill -9, system reboot, or upgrade), running pipelines lose progress and restart from the beginning.

Symptoms

A pipeline that was 90% complete restarts from 0% after an edge restart, with no indication in execution status or logs that state was lost.

How to Check

Check if the edge process recently restarted:

# Check when edge process last started
ps aux | grep expanso-edge

# Check system logs for edge restarts
journalctl -u expanso-edge | grep -E "Started|Stopped|Killed"

# Check edge uptime
systemctl status expanso-edge

If the edge recently restarted and you have long-running executions, those executions likely lost progress.

Workarounds

1. Design pipelines to be idempotent - Make sure reprocessing data from the beginning is safe
2. Use external checkpointing - Store progress in external state (database, file) and resume from there
3. Monitor edge uptime - Set up alerts for edge process restarts
4. Add explicit logging - Log progress markers so you can detect restarts in pipeline logs