Fleet & operations

status

active · ongoing

when

2026-03 → now

came from

Greenfield DevOps platform, SnapDragon-Coder

led to

Marilyn, LiLM / Lucky Labs

The workflows, the backups and the drills that let the lab be rebuilt from zero.

Why it started

Because on 2026-07-18 a local process renamed a parent of my home directory and I found out what was not backed up. See Losing $HOME. Before that, the fleet was a workstation with GPUs. After it, the fleet got the discipline of the day job.

What it is

• Backups as one nightly workflow: datasets to restic on three targets including offsite, the git server, a disaster-recovery kit, and recovery secrets in a managed store. Corpora that a fetch script could not regenerate were the priority.
• A weekly fire drill that rebuilds the git server from the offsite copy on a disposable cloud machine, validates a clone, and pages the phone if it fails or goes silent.
• Break-glass on two continents: stopped cloud instances that can restore the whole fleet from that region’s offsite copies, the second continent because the first shares an earthquake zone with the house. About six dollars a month.
• GPU leases: every card has a lease manager with heartbeat and release. Nothing trains or serves without one.
• A worker restructure: a shared Go library for Temporal workers plus tiny single-purpose workers (backup, lease manager, unit runner, deployer) with a per-file test-coverage bar of 98% and a review that found and fixed seven high-severity issues, including a GPU false-kill.
• Sandboxed workflow units: the home directory is read-only to any job.
• Cloud GPU on demand: a Temporal workflow that rents a GPU from Lambda Cloud, runs the job, and always gives the machine back, with the time-to-live written into the instance name and a reaper that enforces it. Built for the jobs that outgrow a single card. Details in the operating stack.
• A NAS on ZFS, thermal monitoring for drives that had been invisible to it, and network paths that survive a reboot.¹

Where it stands

Active. The Go nightly is the nightly; the Python one is retired. Drills pass weekly in two regions. Open items are mine to schedule: node workers on the other machines, and moving the trainer onto the new lease protocol after v5 lands.

Lineage

• Came from: greenfield platform (k3s, ArgoCD, Temporal were already there), SnapDragon-Coder (first GPU contention)
• Led to: everything that runs unattended now: Marilyn, LiLM

Sub-pages

• The operating stack: every tool, its role, and what it earned its place by
• Local or cloud: how model calls are routed, from Bifrost profiles to the LiteLLM gateway, and why
• Wins · Losses · Rabbit holes