← Research corpus

Losses

Invisible to the counts

• The raw token targets were about eight times inflated. They counted LaTeX and bibliography markup as tokens. Five thousand papers that were supposed to be a paper-heavy tier came to about 40M prose tokens, and the corpus was code-heavy and paper-light. Cost: a re-sizing of v3 on corrected numbers and a decision to drain the whole pool.¹
• A 14.8% blank tier under a report of zero failures. The converter exited cleanly with thousands of near-empty outputs. Nothing in the summary counts could have shown it.²
• A nested-document bug hit 16.2% of papers. Included files that were complete documents brought their own end-of-document markers, and the converter stopped at the first. Found only after the first repair, by four sub-agents reading output. ³
• One paper expanded 277 times: 101 MB from a 0.36 MB source, because a self-referential macro had been harvested from a comment. The repetition rule caught 2 of the 79 affected papers. Twelve papers and 27M tokens of duplicated text had to be excluded. Taught: over-generation is as silent as under-generation, and a size-floor gate waves it through because bigger looks better. ⁴
• A token count was circular. “372M” came from bytes divided by 3.98, and 3.98 had been derived from 372M. ⁵
• Bytes versus string length. Comparing a file size to the length of a decoded string on math-heavy prose fakes a delta of about 9%. Compare bytes to bytes.²

Costs still being paid

• The provenance gap. No tier records the dataset revision it was fetched from, and no original bytes were kept, so tier integrity cannot be re-verified today. The fix is scoped at about an hour and waits behind v5, whose corpus was already pinned and certified when the gap was found; it lands with the next corpus, not by re-opening this one. ⁶
• 78 GB of arXiv source is not backed up, by decision. It is refetchable, but at three seconds per request that is a multi-day re-drain, not a free download, and the notes say so in capitals so it never reads as cheap later. ⁷
• The OCR tier misreads math. A subscript theta became a zero in a spot check, so 627 papers live in a separate tier, flagged by provenance, and never enter math-sensitive evaluation. ⁸
• Two backup mechanisms built in the same hour by different sessions store the derived arXiv prose twice in S3. Reconciled on paper, not yet in the scripts.⁹

Process

• An out-of-memory kill during composition. The composition step is RAM-heavy and was run uncapped on the workstation once. It now runs chunked under a memory limit or on the big box. ¹⁰
• An overlapping tier. The educational-web subset refetched from a larger sample overlaps the earlier small-sample tier by id, so one had to be deduplicated or dropped.¹¹
• A validation split that leaked 100% through the composer for one lineage. Only that lineage’s own held-out set counts. ¹²
• Two converters exited zero having written nothing, one for PDFs and one for LaTeX, weeks apart. See a green exit is not done.