Temporal or it's a POC
If a run cannot survive the death of the session that started it, it is a proof of concept.
The incident
The home-directory wipe of 2026-07-18 killed every Python process I had started from a terminal. Days of work, unrecoverable, because nothing knew it had been running. The jobs that had been submitted as Temporal workflows would have resumed on the next worker.
Later the same rule earned its second half: a result produced outside a workflow cannot be cited or promoted, because nobody can reproduce how it was produced.
The rule, and how it is enforced now
- Every multi-hour job is a Temporal workflow with one activity per unit of work.
- Workflow units run sandboxed: the home directory is read-only to them.
- Training resumes from checkpoints via the workflow, not by hand. A pre-training run of Marilyn v5 survived heartbeat time-outs, a worker restart and a host reboot this way.
- Results pages on this site say which workflow produced the number, or say “unmeasured”.