← Marilyn

Losses

The ones that changed the plan

• v2 was fed about 1.5 tokens per parameter, against the ten or more that sizing rules ask for: under by roughly seven times. Proof #2, the first serious test of whether she could use what she had memorised, scored 0 of 32. Cost: every conclusion drawn from v2, and a re-run still owed once a properly fed model exists. Taught: nothing about a model’s limits is trustworthy until it has been fed.¹
• Identity in the weights overfits. Teaching v2 her own biography by training on it produced a model that recited and could not converse; the drift brake did not catch it because loss went down. Taught: identity belongs in retrieval, not weights, and a brake that watches loss misses overfit by construction. ²
• The imitation ceiling is about 40% accuracy on tool-family items. Three supervised passes of calculator demonstrations: pass one bought the form (the tool fired on most items), passes two and three bought nothing in accuracy (13 to 15 correct of the 36 tool-family items) and cost 11.4% on validation. Cost: three teaching runs and a checkpoint that must not be promoted. Taught: stop passes at the plateau; the next lever is execution reward, which then worked.³
• Tensor-carry is scale-limited at 506M. The D2 experiment that tried to carry state across turns as tensors resolved as limited by model size, not by design. Cost: the experiment; gain: the decision to build a bigger trunk (v5) rather than keep tuning v3.⁴
• Untrained native memory is a clean fail. The Metis-style bolt-on with uniform writes showed nothing above the seed floor beyond the context window on either seed. Cost: one gate run on carlo. Gain: a priced next rung instead of an assumed one, and two side findings: in-context verbatim recall dies between 3,000 and 4,000 tokens, and the paper’s +1 stabiliser produces NaNs untrained. ⁵

Measurement mistakes, caught and recorded

• A corpus number was circular. The arXiv tier’s “372M tokens” came from bytes divided by 3.98, where 3.98 had been derived from 372M. Re-measured by sampling at 411M composed tokens, with an end-to-end check within 0.68%. Taught: derive ratios from independent measurements or not at all. ⁶
• A validation set was 100% contaminated. The arm-D lineage’s shared validation leaked entirely through a composer split, so only its own held-out set counts. Cost: every cross-arm comparison that used it. ²
• Tag hygiene looked like a win and was not. A retrieval-router change scored 15 of 15 on the set it was tuned on and 15 of 23 blind. The ranking work built on it was withdrawn. Taught: verify by attacking the gate.⁷
• A status paragraph was false in four places. The 2026-08-19 summary claimed a test count, a commit, a merge state and a decision count that a later session measured and found wrong. It was replaced, not amended, with each number re-measured.⁸
• Stale clones misled four reviewers during the post-training audit: they read a branch that was hundreds of commits behind and reviewed code that no longer existed.⁹

Infrastructure that bit the science

• A workflow time-out does not kill the unit. A pre-training run’s orchestration timed out and retried while the GPU job kept running, so the sweep burned attempts (4 of 5 used) on a healthy training process. Taught: the orchestrator must own the process, not just the timer. ¹⁰
• Each resume re-packs the corpus for 7.6 hours before a step is trained. Recovering from step 126,000 cost most of a day of the training card doing no learning.¹¹
• The cloud judgement tier ran out of balance overnight and a data build “completed” thousands of rows short. See Local or cloud.¹²
• A shard map lived in a scratch directory and vanished on reboot. Rebuilt durably; every repo script was then grepped for scratch paths.¹³
• A benchmark box ran out of host memory because a serving engine’s prompt cache lives in RAM at about 2.2 GiB per saved 32k-token slot. Serve units now carry a memory cap.¹⁴