Marilyn

status

active · ongoing

when

2026-07 → now

came from

shellm, Research corpus, SnapDragon-Coder

led to

LiLM / Lucky Labs, Bench lane, Paper ingestion & arXiv weekly

A language model trained from random weights on my own hardware, to study why a model that has memorised something still fails to use it.

Why it started

shellm proved a from-scratch model could be trained here, and that its base and corpus were far too small to be useful. The next question was stranger. A small model was re-schooled on study material, demonstrably absorbed it, and then could not route it to answer a question: it deflected. The literature calls this the Knowing–Using Gap (the paper is on Reading that shaped it). That gap is the spine of the project.

What it is

Marilyn begins as random weights. Nothing is distilled from a larger model; when she knows something, she learned it here, and I can point at when. Around her:

• a corpus of about 4.3 billion tokens, assembled and gated (research corpus);
• a nursery: a serve-and-learn service with tools, a checkpoint ring, and a promotion gate that decides whether a new checkpoint is better and safe;
• pre-registered proofs for vocabulary size, weight tying, epochs and model size, each settled by a factorial sweep with seed-pair noise floors before a decision was recorded;
• Temporal workflows for every multi-hour run, so a reboot is an event, not a loss;
• Mantis, a native memory taken from the Metis paper and bolted on as the cheapest possible disproof first. Details on the v5 programme.

Versions: v2 (fed about a seventh of what sizing rules ask for, which taught the sizing lesson), v3 (506M parameters on 4.8B tokens, 0.8575 bits per byte), v4, and v5, a bigger trunk in three variants that are parked as of September 2026 awaiting LiLM’s answer.¹

Who does what

I design the experiments, write the pre-registrations, set the gates and review the results. Agents run the builds, the sweeps and the harvests under those gates, and they are not allowed to promote anything. When a page here says “measured”, the measurement was made by a workflow I can re-run and the verdict was mine.

Held out, and kept that way

A probe set is frozen before training and never used to pick a decision. Consolidation and ingestion pipelines are checked against the frozen sets before they ship, and the corpus was re-certified against every one of them before v5 trained. The rule was earned when one lineage’s validation set leaked entirely through a composer split; only that lineage’s own held-out set counts since. “From scratch” means: random initial weights, a tokenizer trained on my own corpus, a stock transformer block except where Mantis adds memory, and nothing distilled or copied from another model’s outputs or weights.²

Where it stands

The post-training programme (calculator tool, look-up tool, a tensor lane, a knowing-gap diagnostic) is built and its test suites are green; the exit gates are not yet run. The question in the charter is still open. The lab repositories are private; the numbers on these pages are quoted from their records.³

Lineage

• Came from: shellm, research corpus, the RLVR traces of SnapDragon-Coder
• Led to: LiLM (knowledge outside the weights), bench lane, paper ingestion

Sub-pages

• The v5 programme: single trunk, MoE+, MoE+Mantis, and why all three are parked
• Reading that shaped it: the three papers, what was taken, what was found
• Wins · Losses · Rabbit holes