In late October I gave a 30-minute Clubhouse talk on our paper, Forking: Sudden Overfitting Under Replay (arXiv:2610.00394). This is the written English companion to the talk, with the paper’s figures: what over-encoding is, why replaying the same training data forks the loss curves, the explanatory model and the intervention that tests it, three empirical laws, and what it means for training large models in practice.
A standard language model has a token embedding table that maps each token to a vector; each Transformer block then adds a residual on top of it. An over-encoding module adds one more learned representation, for combinations of tokens — an n-gram.
Take “there is a dog”, treating each word as one token: “there is a” is the context and the prediction target is “dog”. Already from this three-word context we know at least one thing: the next token should be a noun. Two-token contexts are bigrams, three-token contexts are trigrams; DeepSeek’s Engram uses tables up to 5-grams.
The module uses the context as a key, retrieves a learned vector from a lookup table, and feeds it into the Transformer — where exactly it enters the block is a design choice.
The appeal is cost. Lookup is cheap compared with running a large dense network; it can be CPU-only, and the table can live in host memory or even on SSD. It looks like a new scaling axis — an axis for memory.
Our paper is not an argument that over-encoding does not work. It works: better prediction, useful extra capacity. What we point out is a failure mode when it is combined with multi-epoch training: very common at the model sizes research uses, less obvious with large data, but it returns when production-scale data meets multi-epoch training.
We found it in the nanoGPT speedrun. We added an n-gram module to the architecture and trained on a multi-epoch dataset, with the same backbone without the module as control. At the start of the second epoch the training loss drops like a staircase while the validation loss diverges upward, like the prongs of a fork — hence forking. In the third epoch the staircase returns and the gap widens again.
This sounds like ordinary overfitting. But LLMs usually do not overfit: the data is too massive and the regularities of natural language too complex, so LLMs normally sit in an underfitting regime. Overfitting now shows up even on extremely large MoEs; over-encoding is similar, except more violent.
Imagine “there is a” appears twice in a small training sample, both times followed by “dog”. In the wider world dog and cat might be equally likely; the sample just happened to contain no cats. The model has seen “cat” many times elsewhere, but never after this phrase.
Now replay the two examples. Every pass rewards the model for predicting “dog”; more probability goes to dog, less is left for cat. On a held-out “there is a cat” the model gets worse even as its training loss improves. Repeating the same examples adds no new evidence: the model becomes more confident without becoming better informed.
n-gram memory sharpens this. Each context carries its own lookup vector, a near-independent small prediction problem; training on other examples does little to fill that hole. Separating seen from unseen continuations makes the squeeze directly visible: the loss on unseen continuations climbs through replay.
An explanation needs more than a story. Mid-training we masked memory retrieval for a subset of contexts and left the rest untouched. The masked contexts largely stopped growing their gap; the unmasked ones kept widening. Change one part of the mechanism and only the corresponding part of the behavior changes — that is why we trust the explanation.
Three experiments, three patterns. First, the gap grows as context frequency falls, close to a power law: around 0.25 for bigrams and 0.33 for trigrams in our early experiments. Second, larger tables give larger gaps: with a small table, different contexts stack onto the same entry, so each vector is effectively visited by more samples. Third, larger corpora give smaller gaps: rare contexts become fewer.
The power law was the hardest to explain. Plain finite-sample overfitting would suggest an exponent of minus one half; we measured clearly flatter. I first guessed training dynamics, then the statistics of natural language — the latter won. The clue was Good-Turing: after an n-gram context has been sampled f times, the probability that the next token is still unseen falls like f to the minus alpha, and that alpha matches our exponents. Following that clue we separated unseen continuations from seen ones: within a context’s own subspace, seen tokens squeeze the probability of unseen ones — the mechanism by which replay pushes the gap wider.
A few rules for real training. Freeze the over-encoding tables during SFT and RL; freezing still leaves some forking, because the context-specific subspaces are already separated and the backbone can keep exploiting them. In SFT, do not repeat similar patterns too many times — many examples is not the same as variety.
More importantly: even if you train for a single epoch, forking deserves some attention. Under one epoch there are really two training losses: the loss on tokens already seen, and the online loss on the fresh tokens of each new batch — which is strictly equal to the validation loss, and is why first-epoch pretraining needs no same-distribution validation set. So the damage does not announce itself as a fork; what you may see is the loss descending a little more slowly than expected. A cheap check at the end of training: fix a set of trained-on examples and measure their loss against untrained tokens. If forking were solved completely, the curve should fall faster.
The public reports for DeepSeek V4.1 and Qwen3.8 Flash Next do not mention the phenomenon. I suspect two reasons: the phases where tables are trainable mostly do single-epoch training; and the tables are far from their scaling boundary — the Engram authors themselves say there is room to scale, and they are scaling: the October release of DeepSeek V4.1 Flash carries a notably larger n-gram table than the September one, roughly 150B to 250B parameters. We are running experiments on how far a DeepSeek-scale table has to grow before this problem appears.
Also note that real pretraining does repeat some specialized data; those are the places where it triggers. My understanding is that foundation-model teams have seen this failure mode in production-scale experiments too, diluted in the overall averages so that a single aggregate curve hides it. Companies have already approached me about collaborating on it.
Finally, the bigger picture. My ambition is to establish a science of AI — SAI: definite derivations between conclusions. It starts as a standard of evidence. My handle on Xiaohongshu, the Red Book, is “Be Critical”: I often feel papers’ conclusions outrun what their experiments establish, or the conditions under which a conclusion holds go unstated — we know what happened but not why, or when it will happen again.
I do not think everything must be derived from pure mathematics; then we could only compute multi-layer linear networks. I believe Anderson: more is different. Complex phenomena resist pure mathematics, but intermediate levels host usable laws that decouple from microscopic detail — the ideal gas law, Newton’s laws — without computing every atom. We can build intermediate first principles: relationships between whole-system quantities that we can isolate, study separately, and connect into further predictions.
A simple picture: suppose we can derive exactly A = B + C, where C cannot be computed but can be measured. If C stays small across the conditions we care about, then A is approximately B — a useful conclusion with an explicit approximation, a measurable error, and a checked range of validity. The second requirement: change B in a controlled way and check that A responds as predicted. Finding A and B equal by accident in one experiment is not enough; the relationship must support prediction under intervention. That is the causal part.
Research and this standard form a cycle: experiments give phenomena, explanations suggest new experiments, new experiments mark where the explanation holds and where it must improve. Forking is an instance of the method — a problem small enough to settle on modest compute: phenomenon, explanatory model, intervention test, three laws, then back to the production-scale question. Limited compute makes choosing the right question and the right comparison more important, not less.
The phenomenon was discovered by Danni Yu and first studied by Shanbin Yu; I joined partway through and built the explanatory model — the part I am proud of. The work was funded by MetaCircle, Ziming Liu’s research company; I will start my PhD at LiuLab.
All talk materials are public here: the English read-aloud script (PDF), with per-section timing checkpoints and speaker notes, and the Q&A reading script (PDF), thirteen questions with my planned answers. The paper: arXiv:2610.00394; over-encoding background: the Engram paper, arXiv:2601.07372.