← LiLM / Lucky Labs

Losses

The honest findings from the real-corpus gate

• The summariser was truncating, not summarising. A 71,000-character document became one “summary” that was its first 500 characters verbatim, plus a fabricated methodology stub. About one node per document instead of many. Cost: the v1 product is a searchable substrate, not a knowledge graph. Taught: check the derivative against the source, not against the schema. ¹
• Relations were about 100× sparser than on synthetic data: 701 dependency edges across 244,242 nodes, 0.29%. The synthetic corpus every earlier spike had used was procedurally generated with one edge type and every node carrying up to two edges, so the graph looked dense because the generator made it so. The knowing-graph the architecture promises does not emerge from real text without a real linker. Decision: v1 ships without it; a linker and real summarisation are v2. ²
• The equation extractor fired on currency. On web and wiki tiers only 9 to 20% of “equations” were LaTeX; $40,000 looked like math. Fixed with a gate; a reminder that a regex tuned on papers is wrong on the web. ³
• Scaffolding polluted every node’s terms, plus an empty-string term on 9.3% of nodes that the index silently dropped. Both fixed; both were invisible to the counts.⁴

Five failed full builds before one succeeded

• Build 1: the shard-ingest activity did not heartbeat on full-size shards, so Temporal timed it out and exhausted retries on a healthy process. Fixed with a heartbeat callback.⁵
• Build 2 caused a fleet incident. Its 233 GB workspace sat on carlo’s root filesystem, which is the one the kubelet monitors. Disk pressure tainted the node, Temporal itself was evicted, and the build died with the orchestrator. Taught, in capitals in the notes: never build on a kubelet-monitored filesystem. The workspace moved to the SATA SSD after I freed about 700 GB. ³
• Build 3 was killed for lack of observability: a parallel pool inside one activity gave no per-shard progress. Replaced by Temporal-native fan-out, one activity per shard.⁶
• Build 4 died on a storage I/O error from a store that had been set up as a plain stripe, chosen for speed on data that is rebuilt from source. Once it started holding work that was the wrong trade, and it was rebuilt with redundancy before the next attempt.⁷
• Build 5 stalled on a database lock. The store’s embedded database kept the default five-second lock timeout; on the slow external pool a write lock or checkpoint held longer than that and the run failed with “database is locked”. Fixed by sizing the timeout to the medium. ⁸
• A worker that cannot interrupt itself. A process-pool Temporal worker can never cancel a timed-out attempt from inside; the attempt runs to completion regardless. A structural limit of that worker shape, recorded so it is not rediscovered.⁹
• A cold run once ingested the wrong corpus: a raw harvest tier instead of the composed one. Caught by comparing record shapes. ³

Gates that were wrong before they were right

• The gate’s tier-prefix-leak rule was a symptom heuristic that false-positived on real ids (an arXiv id legitimately starts with arxiv:). Replaced with an invariant. See gate on invariants. ¹⁰