The v5 programme
v5 was sized from an in-house study rather than a rule of thumb. The decisive measurement: a twelve-cell two-hop composition test, run across twenty checkpoints of the 506M v3 training run, never rose above 3 of 12 at any point in training. A ceiling that does not move with more data is a size limit, not a data limit. The follow-on sizing sweep put the composition floor at depth and width together, at least 28 layers at 1,536 wide, while content precision tracked tokens. The target was 32 layers, d_model 1,536, about 931M parameters, on roughly 40B tokens, about 43 tokens per parameter. Three variants were then built on that trunk, each a separate lane with its own workflows, so that the comparison is between architectures and not between accidents.1
| Variant | What it is | State (2026-09-04) |
|---|---|---|
| Single trunk (dense) | One 931M model, the baseline everything else is measured against. | Pre-trained; recovered from a stall at step 126,000 on 2026-08-29 and landed at the end of August. |
| MoE+ | A shared trunk plus domain experts (code, math, science, web, Go, Python), each pre-trained on its own diet and taught to read from the trunk. Composition is gated with null arms: wrong expert, shuffled read-out. | Trunk seated and admitted 6 of 6; all six experts pre-trained and read-tuned; the composition gate is built and null-arm-compliant, and has not yet been run against a trained composition. |
| MoE+Mantis | MoE+ with the native memory described below, plus the self-care cycle. | Mechanism proven at signal level; self-care dose curve measured.2 |
The numbers, one table
| Version | Parameters | Tokens | Tokens per parameter | Bits per byte | State |
|---|---|---|---|---|---|
| v2 | small | about 1.5 per parameter | ~1.5 | unmeasured on a comparable set | superseded; taught the sizing lesson |
| v3 | 506M | 4.8B | ~10 | 0.8575 | the reference point |
| v4 | 506M-class | more, on the v4 corpus | unmeasured here | unmeasured here | superseded by v5 |
| v5 single trunk | 931M | ~40B | ~43 | unmeasured: exit gates not yet run | landed, parked |
| v5 MoE+ | trunk + six experts | own diets | per expert | unmeasured: composition gate not yet run | built, parked |
| v5 MoE+Mantis | as above + memory | as above | as above | unmeasured | signal-level only, parked |
“Unmeasured” is a statement, not a placeholder: the number does not exist yet on a set comparable to v3’s, and this table will be updated when it does.3
Mantis
Mantis is Marilyn’s native memory: a per-layer fast-weight state written by outer products during the forward pass and read back through linear attention blended into the attention branch. The mechanism is taken directly from the Metis paper (Reading that shaped it); the name is mine. It was bolted onto the frozen v3 weights first, untrained, as the cheapest possible disproof. That run failed cleanly, which was the point: the gap between an untrained bolt-on and the paper’s numbers is exactly what their training buys, and it priced the next rung instead of assuming it. Two side findings came free: v3’s in-context verbatim recall dies between 3,000 and 4,000 tokens even with the fact inside the window, and the paper’s +1 read-denominator stabiliser produces NaNs when untrained. A second version with gated and delta-rule writes followed, then a “self-care” cycle that measured how much stored binding survives repeated healing passes on real text: a dose curve, not a yes.4
Why all three are parked
As of September 2026, none of the three variants is training. The reason is the charter’s own rule, explore freely and gate investment: LiLM asks whether a model can learn, self-learn and be patched without a training run, by holding knowledge outside the weights where it can be ingested, minted and corrected with provenance. All three variants are ways of holding more inside. LiLM’s answer decides which variant, if any, gets the next tranche of GPU time.5
Sources
Numbers on this page are quoted from the lab's own records. The records are private; each note gives the record's date.