← Research corpus

Rabbit holes

Where the corpus came from

• A deep-learning specialist, by continued pre-training of an existing model. The original plan (2026-07-22): 889 papers, 875 flattened LaTeX sources and 15 textbooks as the nucleus, a Qwen-class base continued on it, then supervised tuning on paper-anchored derivations. A 1.5B baseline was measured (validation loss and a small ML exam). Dead end as the vehicle: the three turns went from-scratch instead. Alive as arm C, the comparison lane, and as the first corpus. ¹
• Research digests as the curation layer. Two hundred papers summarised with relevance labels, intended as curriculum weights and fine-tuning seeds. Became the reading habit that turned into paper ingestion. ²
• The citation graph as a selection method. Tier 3 of the first corpus followed citations outward from seed papers to 681 canon papers. Its blind spot was named at the time: citation graphs never surface papers newer than the seeds. Became the nightly ingestion of new papers, which fixes exactly that. ³

Detours in building it

• Textbooks by OCR. Fifteen textbooks with recorded licences, deferred because OCR needs a GPU and misreads math. Still deferred. ⁴
• A CUDA source tier. 425M tokens staged; keep-or-drop was open for weeks. Decided by the v5 tier list rather than on its own merits. ³
• Structural recovery when the parser refuses. A tool that salvages prose before a parse error, provided the cut lands at environment depth zero. Built and validated, never put in the pipeline: the repair made it unnecessary. ⁵
• A faster OCR tool, four version pins deep. Unmaintained since 2023, needed four pinned dependencies to run, and before the last pin exited successfully having written nothing. Dead end; the slower tool won on fidelity. ⁶
• Keep or drop a web tier. The essential-web tier was debated and kept; recorded as settled so it is not re-litigated. ⁷
• Provenance at fetch time. Once no tier could prove which dataset revision it came from, the fix was scoped: record the dataset commit and the output hash when fetching. Queued behind the v5 corpus, which was already pinned.⁸

Detours that became the standard

• A hand-made restic tag for two unprotected corpora, the evening they were found unprotected. Became the pattern the nightly adopted. ⁹
• A quality gate written to catch one bug became the gate every stage passes: reject below a floor, warn below a threshold, reject above a ceiling, on a prose-eligible body with figures and code stripped first. ⁵
• The corpus as a retrieval substrate. The same composed shards that train Marilyn now feed LiLM, the project that may decide whether she needs to hold them in her weights at all.