← Fleet & operations

The operating stack

Every tool here was adopted because something went wrong without it. The table says what each one does for the fleet; the notes say what it earned its place by.

Disaster recovery, in three rings

1. Nightly. One Temporal workflow: datasets → restic on three targets → git server bundle → DR kit → recovery secrets → offsite mirror. A worker written in Go now runs it, after a Python one ran in parallel for enough green nights to earn the cutover.
2. Weekly fire drill. A Temporal schedule launches a disposable EC2 box, which pulls nothing but the offsite backups and the managed-store secrets, rebuilds the git server, clones from it to validate, and tears itself down. Any failed step is a typed workflow failure, and Grafana pages the phone on failure or on silence. A green run takes about 200 seconds end to end, of which the restore itself is about 130; a sub-second completion is itself a failure signature. Cost per run is about a cent.²
3. Break-glass, two continents. Standing, stopped instances in two regions, each able to restore the fleet from the offsite copies in its own region: start the node, its role fetches the secrets and the backups, restore. No secret ever rests on the idle volume. The second region is on another continent because the first shares an earthquake zone with the house. All-in cost about six dollars a month. The second-region restore was rehearsed on 2026-08-15 on the standing node itself, from a browser session: start the node, grow its disk, refresh the kit, run the same restore script the weekly drill runs, and pass three checks (restore complete, repositories visible through a real admin login from the region’s own secrets, a clone on the box). The rehearsal found three things that would have failed live despite green unit tests, a restore script hard-wired to the home region’s bucket, a node with no container runtime, and a disk too small for the restore, and all three were fixed before it went green. Wall time was not recorded; the restore step is the same one the drill completes in about 130 seconds, so recovery time is set by how fast a human starts the node, and the data-loss window by the nightly backup plus replication lag. The next rehearsal will record the clock.³

Cloud GPU on demand: rent, run, always return

The fleet’s ceiling is a single workstation card. Anything bigger, a 1.5 TB teacher model in bf16, a large pruning calibration, teacher-scale distillation, needs rented GPUs. The pattern was built on 2026-07-18 on the cheapest sandbox available (a single A10 at $1.29 an hour) so that swapping one field scales the same workflow to real rigs.⁴

The shape. One Temporal workflow per job, five provider activities: provision, await ready, run the commands over ssh, terminate, verify gone. A second provider is a new activities module, not a new workflow.

“Never keep paying”, three independent layers, because a leaked instance is the failure mode that costs real money:

1. Termination runs in a finally block with stubborn retries, so a failed job still returns its machine.
2. The time-to-live is encoded in the instance name itself. A reaper workflow sweeps every instance the fleet owns and kills anything past its encoded expiry. This covers the case where the worker or the workflow itself dies.
3. Everything the fleet rents matches one name prefix, so a single API call is the audit.

Region policy. Availability first, anywhere in North America; the west coast is a tiebreak only, because egress is free and most single-GPU types show zero capacity most of the time. Pass a list of regions and take the first with stock.

Maturity, item by item. Proven live: an end-to-end hello (provision, about seven minutes to boot, job, terminate verified, account back at zero instances, about 30 cents); a model-serving smoke (fresh machine, Ollama installed, a 7B model pulled and queried through an OpenAI-compatible endpoint, torn down); and the staged pattern (download big weights once to a persistent volume, serve from them later on a different machine, with idempotent re-attachment). Specified and reviewed, not yet run: a resumable multi-day training job with the checkpoint on a persistent volume and an offsite sync that runs even on failure. Findings along the way included the provider’s API rejecting Python’s default user-agent string, which presents as a misleading 403 on a valid key.⁵

Provider credentials never travel in a workflow payload, the same rule as every other secret on the fleet.

What the stack is not

CI is paused while the fleet is in training mode; merges are gated by a clean-clone check instead. Written down because “the check is paused” is a different state from “the check passed”, and the site should not blur them.