OPHIS n-gram gap — Agent Experiment Log
Machine-oriented index for agents. Human report: ngram-gap-mechanism-guide.html ·
partner bridge: ngram-gap-regime-bridge.html · task spec: manual.md.
Keep this terse. Append new experiments under §Log using the entry template. Last updated 2026-07-26.
Claim under investigation
n-gram–augmented nanoGPT: at epoch2/3, train loss steps down while val loss lifts → train/val gap.
Hypothesis: n-gram value tables overfit rare contexts — after repeated exposure the model learns a shortcut on
train-specific n-gram patterns; because the rare-context train/val distribution differs, val lifts.
Mechanism chain (measured): new epoch → n-gram gate activates (opens injection channel) → table content accumulates
train-specific pattern → gated injection RMS grows → gap. Writing angle: over-encoded n-gram params should be frozen
in multi-epoch post-training (SFT/RL) to avoid this overfit.
Key findings (indexed)
| id | finding | evidence |
|---|
| F1 | Gate/table norm growth is time-aligned with gap onset (current-shell positive). | guide §11.9; obth-history10 obcurves |
| F2 | table RMSProp (nanogpt_original): epoch2 no fork, epoch3 fork. Table content RMS jumps at gap onset (step≈783), aligned. | guide §11.10 B.1; figB1_rmsprop_norms.svg, figB3_rmsprop_alignment.svg |
| F3 | + Muon (nanogpt_original): delayed gap (onset≈647, late epoch2). Injection norm builds epoch1–2, gap peaks epoch3 → 11.9-style time-lag. | guide §11.10 B.2; figB1_muon_norms.svg, figB3_muon_alignment.svg |
| F4 | Gap is driven by mid-frequency n-gram contexts (21-50 … 201-500 hit buckets), not ultra-high-freq (5k+ fits both train&val). | figB2_*_freqloss_*.svg; guide §14.7 |
| F5 | gate/gated norm observables ARE recorded for nanogpt_original (not only current-shell): NanoGPTSelfAttention._add_value_residual calls record_gate + record_branch_readout. | train.py ~L1721 (bottom-up branch) |
| F6 | Negative control (table AdamW, matrix AdamW, no RMSProp/Muon): no sustained gap onset detected, max per-epoch gap ≈0.38 (vs final-step gap 0.6 for RMSProp, 3.2 for Muon). Final-step gap only 0.054 (4.558 train / 4.612 val) vs 0.599 (RMSProp) / 3.233 (Muon). Confirms gap requires an optimizer asymmetry (table RMSProp or matrix Muon), not just having n-gram tables present. | guide §11.10 B.0; figB1_nofork_norms.svg, figB4_nofork_alignment_interactive.html |
| F7 | [UPDATED fixv3] Freeze ablation: freezing n-gram tables + injection gates (requires_grad_(False)) right after epoch1 suppresses per-epoch max gap from 0.688 (baseline rmsprop) to 0.245 (frozen) — roughly to 1/3, not as dramatic as an earlier fixv2-era estimate of 1/4–1/10 (that estimate used the since-superseded online-cumulative-counter frequency methodology; direction of the effect is unchanged, magnitude is smaller under the corrected fully-offline methodology). Confirms n-gram table/gate updates during epoch2/3 are a primary gap driver, though not as completely dominant as first estimated. | guide §11.11 C.1; figB6_freeze_ablation_gap.svg |
| F8 | [UPDATED fixv3] More training data (counter-hypothesis result): doubling train shards (1→2, step budget scaled 1000→2200 to still reach epoch3/4) does NOT shrink the gap — max per-epoch gap grows from 0.688 (baseline, 3 epochs) to 1.385 (moredata, 4 epochs). Repeated exposure across more epochs still appears to deepen overfitting rather than dilute it, confirming the fixv2-era finding under the corrected offline-frequency methodology. Still contradicts the original hypothesis and still needs a follow-up experiment with genuinely larger/more diverse corpus (not just more epochs over the same doc set) to test the intended claim. | guide §11.11 C.2; figB7_moredata_gap.svg |
| F9 | [UPDATED fixv3] Epoch3 gap-vs-frequency curve (from the fully-offline context_hash/offline_freq_count raw per-token probe log): both positive settings show gap decreasing with increasing context frequency — rmsprop starts ≈1.28 at freq≈25 and falls to ≈0.26–0.4 by high freq (with some bucket noise, e.g. a dip to −0.047 at freq≈10801); Muon starts higher (≈3.68 at freq≈25) and also decays with frequency. Reproduces F4's mid/low-frequency-driven gap shape using the corrected, fully offline per-context frequency data (supersedes the fixv2-era numbers, which used the online-cumulative-counter methodology). | guide §11.11 C.3; figB5_epoch3_gapvsfreq_trigram.svg |
Runs
Active reruns (this task, seed42, 1000 step, single-shard tr1, nanogpt_original — 3 settings incl. negative control)
run dir (local remote_training_runs/) | ngram opt | matrix opt | gap onset | final train/val | orig train/val | target gap |
|---|
| 20260726_nano_adamw_adamw_111_full_tr1_1000step_seed42_normobs | adamw | adamw | none (no sustained onset) | 4.558 / 4.612 | — | negative control, no target gap |
| 20260725_nano_rmsprop_111_full_tr1_1000step_seed42_normobs | rmsprop | adamw | step 783 (e3) | 3.707 / 4.306 | 3.587 / 4.317 | delayed→e3 fork |
| 20260725_nano_adamw_matrixmuon_111_full_tr1_1000step_seed42_normobs | adamw | muon | step 647 (e2) | 1.381 / 4.614 | 1.492 / 4.534 | delayed gap |
Each dir has observable_curves.obcurves.json + train.log. Orig values from regime-bridge curves nano-rmsprop/nano-muon/nano-full. Epoch bounds: step 337, 686. nofork orig loss values not committed anywhere (regime-bridge's nano-full curve was loss-only, not fetched to a local run dir before this task).
3rd rerun (nofork, negative control) launched 2026-07-25 reusing same staged code/work dir; see §Log.
fixv2 reruns (2026-07-25, epoch-local frequency freeze + raw per-token probe log + freeze/moredata ablations)
run dir (local remote_training_runs/) | purpose | gap onset | per-epoch max gap | final train/val |
|---|
| 20260725d_nano_adamw_adamw_nofork_fixv2 | negative control, refreshed w/ fixed capture | none | e1 0.52, e2 0.16, e3 0.20 | 4.56 / 4.61 (approx) |
| 20260725d_nano_rmsprop_fixv2 | table RMSProp, refreshed w/ fixed capture (new baseline for C.1/C.2) | step 782 (e3) | e1 0.43, e2 0.39, e3 0.65 | 3.66 / 4.31 |
| 20260725d_nano_muon_fixv2 | + Muon, refreshed w/ fixed capture | step 645 (e2) | e1 0.62, e2 1.38, e3 3.22 | 1.40 / 4.62 |
| 20260725d_nano_rmsprop_freeze_fixv2 | new: freeze n-gram tables+gates after epoch1 (FREEZE_NGRAM_AND_GATE_AFTER_EPOCH1=1) | none | e1 0.49, e2 0.11, e3 0.065 | 5.03 / 5.07 |
| 20260725d_nano_rmsprop_moredata_fixv2 | new: 2 shards, 2200 steps (vs 1 shard/1000 for baseline) | step 1553 (e3) | e1 0.46, e2 0.38, e3 1.23, e4 1.50 | 2.51 / 4.01 |
All 5 use the patched tmp/norm_verify_stage/train.py (epoch-local hit-count freeze fixing the G5/cumulative-counter bug, plus ngram_raw_probe_log.jsonl raw per-token+per-layer-norm logging). freeze/moredata analyzed in guide §11.11 (C.1/C.2); gap-vs-freq curve (C.3) built from the 3 base fixv2 settings' raw probe logs. See §Gotchas G6 for the epoch-indexing bug this fix required, and §Log for the freeze-verification method. Superseded by fixv3 below — the epoch-frozen cumulative counter used here was itself found to have timing/race issues; do not use these fixv2 numbers for further analysis.
fixv3 reruns (2026-07-25, supersedes fixv2 — fully offline frequency computation, per-token/per-context raw schema)
run dir (local remote_training_runs/) | purpose | gap onset | per-epoch max gap | notes |
|---|
| 20260725e_nano_adamw_adamw_nofork_fixv3 | negative control | step 932 (small late-e3 fork, real but minor — see §Log) | max 0.265 overall | 1000 pts, 1850 series |
| 20260725e_nano_rmsprop_fixv3 | table RMSProp baseline | step 801 (e3) | max 0.688 overall | 1000 pts, 1738 series |
| 20260725e_nano_muon_fixv3 | + Muon | step 647 (e2) | — | 1000 pts, 1850 series |
| 20260725e_nano_rmsprop_freeze_fixv3 | freeze n-gram tables+gates after epoch1 | none | max 0.245 overall (vs 0.688 unfrozen) | 945 loss pts (steps ~119-173 unreadable in train.log due to a transient double-write corruption from an accidental duplicate process launch; obcurves/raw-log data for those steps are intact) |
| 20260725e_nano_rmsprop_moredata_fixv3 | 2 shards, 2200 steps | step 1318 (e4) | max 1.385 overall (vs 0.688 baseline) | 2200 loss pts, epochs=[686,1384,2082] |
Fully offline, training-loop-decoupled frequency computation (NGRAM_HIT_OFFLINE_COUNT, via compute_offline_ngram_hit_counts() run once before training under TRAIN_DATA_MODE=fixed determinism) — supersedes fixv2's epoch-frozen but still-online NGRAM_HIT_CUM_COUNT. Raw log schema per line: branch, context_hash, epoch, loss_sum, offline_freq_count, per_token_loss, split, step, tok_count (was freq_count in fixv2). See §Gotchas G7 for the two bugs fixed to get these runs clean (device-mismatch crash, disk-exhaustion crash), and §Log for the fixv3 writeup. These 5 dirs are the current canonical source for all further gap/frequency analysis.
Intervention runs (2026-07-30, guide §10 exp4/exp5 — causal gap-elimination verification)
run dir (server /data3/guoshaoyang/ngram-gap-exp/runs/) | shell | intervention | gap onset | per-epoch max gap (e1/e2/e3) | final train/val (gap) | vs baseline final |
|---|
| exp4_hashreseed_nanogpt | nanogpt_original | per-epoch hash reseed (NGRAM_HASH_RESEED_PER_EPOCH=1) | none sustained | 0.243 / 0.257 / 0.371 | 4.189 / 4.237 (0.047) | −93% (0.688→0.047) |
| exp4_hashreseed_current | current-shell | per-epoch hash reseed | none sustained | 0.571 / 0.307 / 0.351 | 3.885 / 4.137 (0.252) | −93% (3.658→0.252) |
| exp5_lowfreq_gatezero_nanogpt | nanogpt_original | low-freq gate=0 (NGRAM_LOWFREQ_GATE_ZERO=1, ranges bigram:1-200;trigram:1-200) | none sustained | 0.368 / 0.166 / 0.102 | 4.760 / 4.800 (0.040) | −94% (0.688→0.040) |
| exp5_lowfreq_gatezero_current | current-shell | low-freq gate=0 | step ~700 (e3, attenuated) | 0.614 / 0.543 / 1.523 | 3.087 / 4.611 (1.523) | −58% (3.658→1.523) |
All 4 run on ophis-gpu (H200, GPU 1) via /data3/guoshaoyang/ngram-gap-exp/run_exp.sh, same B.1 config as 20260725e_nano_rmsprop_fixv3 (table RMSProp, seed42, 1000 step, 1 shard tr1, vocab=8192, n_layer=8) + the two-shell variants. Baselines: nanogpt_original = 20260725e_nano_rmsprop_fixv3 (final gap 0.688); current-shell = 20260729-allgram-mask-baseline (final gap 3.658). Source train.py = patched copy of temp-20260729-allgram-mask-baseline/train.py (added NGRAM_HASH_RESEED_PER_EPOCH + _reseed_ngram_hash_primes for exp4; NGRAM_LOWFREQ_GATE_ZERO + _set_lowfreq_gate_mask/gate-zero in _add_value_residual for exp5; defensive .get() on ngram_table_beta2_initial for the warmdown block). Conclusion: exp4 eliminates gap on both shells (~−93%); exp5 eliminates on nanogpt_original (−94%) and attenuates on current-shell (−58%, low-freq alone insufficient there). See guide §10 for the full writeup + compound figure.
Fine-grained frequency decomposition run (2026-07-31, guide §11 exp6)
| run dir (server) | shell | intervention | steps | epochs | final train/val (gap) | note |
|---|
| exp6_freqdecomp_current | current-shell | none (observation only) | 1700 | 5 | 1.382 / 5.829 (4.447) | fine-grained bucketing: NGRAM_GLOBAL_FREQUENCY_BUCKET_EDGES=0,1,2,3,4,5,6,7,8,9,10,15,21,31,51,101,201,501,1001,5001 (per-repeat-count resolution 1-9). Same config as baseline_current but extended to 5 epochs. Gap grows from 0.04 (e2) → 0.70 (e3) → 2.47 (e4) → 4.45 (e5). Key finding: low-freq buckets gap first (e2: bucket "1" gap≈1.12, 5k+ gap≈0.02), but by e5 all buckets have large gap (bucket "1" gap≈6.70, 5k+ gap≈5.37). |
Run on ophis-gpu (H200, GPU 6). Source: patched train.py with NGRAM_GLOBAL_FREQUENCY_BUCKET_EDGES env override (makes the bucket edges configurable; default unchanged). Artifacts: allgram_frequency_decomposition.jsonl (3840 records = 20 buckets × 2 branches × 48 probe steps × 2 probe sets), global_frequency_probe_observations.jsonl, observable_curves.obcurves.json. Local copy in remote_training_runs/exp6_freqdecomp_current/. Build tool: tools/build_exp6_freq_gap.py; render: tools/render_exp6_freq_gap.py → docs/interactive/fig_exp6_freq_gap.html + docs/figs/fig_exp6_freq_gap.json. See guide §11 for the interactive figure + numerical table + key findings.
Reference run (norm/gate + hitbucket source for current-shell positive)
Local ../observable_curves_history10.json (4.5MB) = run 20260721-min-bigram-trigram-obth-history10 (current-shell positive, arch nanogpt_current_shell + Muon + mixed opt). Source of guide §10.2/§11 gate charts.
Prior matrix runs (branch origin/windows/bottom-up-gap-decomposition)
Full arch-vs-gap matrix: architecture_gap_experiment_matrix.md. Committed run artifacts under remote_training_runs/<name>/fetched/ (train.log, run_meta.json) — 22 current-shell ablations (20260720-*, 20260721-*). The two nanogpt_original settings above were NOT committed there; regenerated in this task.
Observable series (obcurves .series keys)
obcurves schema: {records:[{step,epoch,values}], series:{name:[{step,value}]}}. Enable via THEORY_OBS_CAPTURE=1 (+ compact/direct_kernel/row_history/history_ablation). Key series:
| group | series pattern | meaning |
|---|
| gate norm | theory.reader_compact.{unigram,bigram,trigram}.gate_rms / .gate_mean | per-order injection gate magnitude |
| table content | theory.reader_compact.{...}.raw_value_rms | table content RMS before gate |
| injection | theory.reader_compact.{...}.gated_value_rms | gate×table = actual injected RMS |
| table coverage | theory.rows_compact.{bigram,trigram}.{table_coverage,entropy_norm} | row usage/entropy |
| order-ablation loss | theory.mask_loss.{train_early,train_middle,train_late,val_fixed}.{normal,no_bigram,no_trigram,no_high_order} | loss with an n-gram order masked |
| hit-count buckets new | theory.hitbucket.{bigram,trigram}.{train,val,gap}.<bin>.{per_token,frac} | per-context-frequency loss; bins novel,1-5,…,5k+ |
| loss/gap | raw_train_loss, val_loss, train_val_gap, epoch_val_bpb | base curves |
| path | purpose |
|---|
OPHIS_gap/tools/extract_norm_alignment.py | Parse train.log+obcurves → figB SVGs, interactive HTML, figB_alignment.json, figB_section.html. Add a run to its RUNS dict + rerun to regenerate. Read-only inputs. |
tmp/norm_verify_stage/train.py | Staged from bottom-up branch + patched with NGRAM_HIT_BUCKET_CAPTURE (bi/trigram context hit histogram + per-bucket train/val/gap loss), NGRAM_RAW_PROBE_LOG_CAPTURE (raw per-token loss + per-layer table/gate norms → ngram_raw_probe_log.jsonl), and FREEZE_NGRAM_AND_GATE_AFTER_EPOCH1 (ablation switch). fixv3 (2026-07-25): frequency source switched to a fully offline, training-loop-decoupled NGRAM_HIT_OFFLINE_COUNT computed once pre-training via compute_offline_ngram_hit_counts() (supersedes the fixv2 epoch-frozen-but-still-online NGRAM_HIT_CUM_COUNT); raw log rows keyed by context_hash/offline_freq_count (was freq_count). New env var NGRAM_RAW_PROBE_LOG_STEPS (default 320,337,338,350,400,450,500,600,686,1000) gates raw-log *writes* to a sparse milestone-step set, independent of the coarser THEORY_OBS_INTERVAL_STEPS aggregate cadence — see G7. Not committed to main tree; pushed to remote work dir. |
Reproduce a run (remote H200)
ssh -p 50002 user@223.167.85.180 # host zp-nc71, 8×H200, venv /home/user/ph/autoresearch/.venv
# work dir: /home/user/ai4ai/ngram_norm_verify_20260725 (staged train.py/observable.py/lib.py/prepare.py/
# gap_experiment.py/token_probe.py + data_split.json={"train":[1],"test":[2..10,6542]})
# cache: dedicated AUTORESEARCH_CACHE_DIR with data/ symlinked to shared shards.
# NOTE lib.py uses /data (a symlink farm) + /data/tokenizer when /data exists — do not retrain tokenizer.
# env (per run): ARCH_VARIANT=nanogpt_original NANOGPT_ENABLE_NGRAM_VE=1 ENABLE_{UNI,BI,TRI}GRAM_VE=1
# NANOGPT_NGRAM_OPTIMIZER={rmsprop|adamw} NANOGPT_MATRIX_OPTIMIZER={adamw|muon}
# POSITION_ENCODING=learned_abs WINDOW_PATTERN=TTTL NANOGPT_ADAM_LR=0.004
# CURRENT_NORMALIZATION=layernorm CURRENT_EMBEDDING_TYING=tied CURRENT_EMBEDDING_INIT=nanogpt_like # << see Gotcha G1
# THEORY_OBS_CAPTURE=1 THEORY_OBS_COMPACT_CAPTURE=1 THEORY_DIRECT_KERNEL_CAPTURE=1
# THEORY_ROW_HISTORY_CAPTURE=1 THEORY_HISTORY_ABLATION_CAPTURE=1
# THEORY_OBS_STEPS=320,337,338,350,400,450,500,600,686 NGRAM_HIT_BUCKET_CAPTURE=1
# MAX_TRAINING_STEPS=1000 SEED=42 TRAIN_DATA_SEED=42 TRAIN_DATA_MODE=fixed REMOTE_RESULT_DIR=<dir>
# fetch back: observable_curves.obcurves.json + train.log ; then run extract_norm_alignment.py
Gotchas
| id | trap |
|---|
| G1 | Config leak: train.py build_model_config reads CURRENT_NORMALIZATION/CURRENT_EMBEDDING_TYING/CURRENT_EMBEDDING_INIT even for nanogpt_original. Their defaults are current-shell (rmsnorm/untied/current) → model barely learns (train stuck ~7.2). MUST set layernorm/tied/nanogpt_like (ps1 runner defaults). Confirm via train.log "Model config" line. |
| G2 | Stale obcurves: completion checks that ls observable_curves.obcurves.json succeed can match a PRIOR run's file. Delete stale obcurves before relaunch, or compare mtime/step count. |
| G3 | The two nanogpt_original settings' norm/hitbucket data were NEVER recorded originally (only loss in regime-bridge). Must rerun to get norms; can't recover from committed artifacts. |
| G4 | hit-bucket capture needs the patched staged train.py (NGRAM_HIT_BUCKET_CAPTURE); the committed branch train.py lacks it. fig14 (guide §14) came from a separate uncommitted hit-tracking run. |
| G5 | [CORRECTED 2026-07-25] Freqloss truncation was a symptom of a deeper methodology bug, now fixed: theory.hitbucket.{branch}.gap.<bin>.per_token lines used to stop early for low-frequency bins because NGRAM_HIT_CUM_COUNT was a full-training-run cumulative counter — the same probe token drifts into higher bins over time as its count keeps growing, so comparing "bin X at step S" vs "bin X at step S'" was comparing different token populations, not just a truncated line. Fixed by freezing the counter at the epoch1→epoch2 boundary (see G6) so a bin refers to a fixed token population for the rest of training; the fix also added ngram_raw_probe_log.jsonl (raw per-token loss + per-layer norms) so downstream bucketing is fully decoupled from the bin edges baked into capture. svg_line_chart's end-marker-circle + footnote logic is kept as a defensive display of any remaining early-truncation, but with the fix in place none of the 3 refreshed baseline runs (nofork/rmsprop/muon fixv2) show truncation anymore. |
| G6 | Epoch is 1-indexed, not 0-indexed: the train dataloader's epoch variable (from x, y, epoch = next(train_loader)) starts at 1, and obs_last_epoch is initialized to this same starting value before the training loop. The epoch-transition hook if trained_epoch != obs_last_epoch: only fires on an actual change (1→2, 2→3, ...) — so a trigger condition of if trained_epoch == 1: (meant to mean "epoch 1 just finished") can NEVER fire, since the loop starts already at 1 and the first transition lands on 2. Correct condition: if trained_epoch == 2:. This silently broke both the hit-count freeze and the FREEZE_NGRAM_AND_GATE_AFTER_EPOCH1 ablation in the first batch of 5 H200 fixv2-precursor runs (all completed without error, but ngram_hit_count_frozen: [] in every summary) — they had to be discarded and rerun. Verify any epoch-boundary hook fires by checking BOTH confirmation log lines ([ngram_hitbucket] froze hit counts... and [freeze_ngram_and_gate] step ...: froze N tensors...) appear at the expected step, not just that the run completed cleanly. Also: train.log's step-progress lines are \r-delimited in places, not \n — pipe through tr '\r' '\n' before grep or matches silently disappear. |
| G7 | [NEW 2026-07-25] Even the epoch-frozen counter (G5's fix) wasn't stable enough, and raw-log write cadence can exhaust disk: two more issues found while producing the fixv3 batch. (1) G5's fix (freeze NGRAM_HIT_CUM_COUNT at the epoch1→2 boundary) was still an online accumulator — subject to timing/race issues around exactly when the freeze fires relative to in-flight scatter_add_ calls. Replaced with a fully offline, training-loop-decoupled NGRAM_HIT_OFFLINE_COUNT, computed once via compute_offline_ngram_hit_counts() before training starts (under TRAIN_DATA_MODE=fixed determinism) — no runtime freeze needed since the table never changes during training. Raw log rows now key off context_hash/offline_freq_count instead of the online freq_count. (2) A device-mismatch crash in record_ngram_hit_bucket_loss: counter.index_select(0, uniq) needs uniq.to(counter.device), since uniq gets .cpu()'d elsewhere but counter/NGRAM_HIT_OFFLINE_COUNT stays on CUDA. (3) A disk-exhaustion crash: raw per-context log writes at the same cadence as the coarse theory-observable aggregates (every ~10 steps via THEORY_OBS_INTERVAL_STEPS) produce 100-400K rows per call, hitting 5-10GB/run and crashing the 438GB remote disk. Fixed with a new NGRAM_RAW_PROBE_LOG_STEPS env var (default 320,337,338,350,400,450,500,600,686,1000, widened for the moredata run's 2200-step budget) + a raw_log_active gate that restricts raw-log writes to that sparse milestone set, while leaving the coarser hitbucket aggregate metrics on the full cadence. Lesson: always rate-limit raw per-context/per-token log writes independently from aggregate-metric capture cadence. |
Doc map
| file | role |
|---|
ngram-gap-mechanism-guide.html | Human report. §11.9 time-lag (current-shell); §11.10 = this task (B.0 negative control / B.1 RMSProp / B.2 Muon alignment); §14 hit-count. |
ngram-gap-regime-bridge.html | Partner's arch-bridge; embeds all runs' loss curves incl. nano-rmsprop/nano-muon/nano-full. |
figB1_<key>_norms.svg | gate/table-content/gated norm per order + epoch/onset markers. |
figB2_<key>_freqloss_{bigram,trigram}.svg | gap by context hit-count bucket; circle marker + footnote where a bin's curve truncates early (G5). |
figB3_<key>_alignment.svg | twin-axis norm-vs-gap time alignment (static thumbnail; trigram gated RMS vs gap only). |
figB4_<key>_alignment_interactive.html | new interactive twin-axis panel: toggle bigram/trigram × gate_rms/raw_value_rms/gated_value_rms × train/val/gap loss overlay. |
figB_<key>_interactive.html | toggle train/val/gap × per-token/total-contribution (frac×loss) × bigram/trigram; view now labels the active metric's units explicitly to distinguish per-token vs contrib. |
figB_alignment.json | machine-readable: per-epoch norm change-rates, gap, datasets. |
figB_section.html | scratch intermediate (the §11.10 block) emitted by the tool; already merged into the guide — do not embed separately. |
figB5_epoch3_gapvsfreq_{branch}.svg | new epoch-3 avg gap (val-train) vs n-gram context hit frequency, log-x, 3 settings overlaid (B.0/B.1/B.2); built from ngram_raw_probe_log.jsonl, bin-edge-decoupled. |
figB6_freeze_ablation_gap.svg | new gap-curve overlay: rmsprop_fixv3 (baseline) vs rmsprop_freeze_fixv3 (n-gram tables+gates frozen after epoch1). |
figB7_moredata_gap.svg | new gap-curve overlay: rmsprop_fixv3 (1 shard) vs rmsprop_moredata_fixv3 (2 shards, epoch-aligned x-axis since step budgets differ). |
figB_extra_analysis.json | new machine-readable: gap-vs-freq curves + freeze/moredata pair-comparison raw points, keyed by analysis name. |
Log
Newest first. Entry template: <div class="entry"><b>YYYY-MM-DD — title</b><br>what/why · runs · outputs · result</div>
2026-07-25 — fixv3: fully offline frequency table + raw-log milestone gating (supersedes fixv2)
What/why: further review found that fixv2's fix (freezing NGRAM_HIT_CUM_COUNT at the epoch1→2 boundary, G5) was still an online accumulator subject to timing/race issues around exactly when the freeze fires relative to in-flight scatter_add_ calls. Replaced with a fully offline, training-loop-decoupled NGRAM_HIT_OFFLINE_COUNT, computed once via compute_offline_ngram_hit_counts() before training starts under TRAIN_DATA_MODE=fixed determinism — the table never changes during training, so there's nothing to freeze. Raw log schema switched from an aggregation-key freq_count to per-token/per-context rows keyed by context_hash/offline_freq_count.
Runs: 5 H200 reruns on the further-patched staged train.py, hit two new bugs mid-run (see G7): a device-mismatch crash in record_ngram_hit_bucket_loss (counter.index_select(0, uniq) needed uniq.to(counter.device)), fixed; and a disk-exhaustion crash from raw-log writes firing at the full ~10-step aggregate-metric cadence (100-400K rows/call, 5-10GB/run against a 438GB disk), fixed by adding NGRAM_RAW_PROBE_LOG_STEPS (sparse milestone-step allowlist) + a raw_log_active gate restricting raw-log writes to that set. All 5 completed to target step with zero tracebacks: 20260725e_nano_{adamw_adamw_nofork,rmsprop,muon,rmsprop_freeze,rmsprop_moredata}_fixv3. One self-caused incident: an accidental duplicate process launch briefly double-wrote to rmsprop_freeze's train.log, corrupting a 10,828-byte span (steps ~119-173) to NUL bytes — confirmed isolated to the human-readable train.log only (obcurves/raw-probe-log data for the same run are intact and unaffected), so it costs 55 of ~1000 loss-curve points for that run but doesn't affect epoch-boundary/gap-onset detection (epoch1 boundary is step 337, past the corrupted window).
Outputs: extract_norm_alignment.py RUNS dict repointed to the 5 fixv3 dirs; load_epoch_freq_gap updated to key off offline_freq_count instead of freq_count (no other function needed changes — the hitbucket/reader_compact obcurves series consumed by the norm/alignment figures are unaffected by the raw-log schema change). Regenerated all figB* outputs against the fixv3 data.
Result: qualitative conclusions from fixv2 hold, but exact numbers shifted meaningfully in one case — F7 freeze-ablation suppression is ~1/3 (0.688→0.245 max per-epoch gap), smaller than the fixv2-era ~1/4–1/10 estimate (the direction/conclusion is unchanged: freezing still substantially suppresses gap, just not as completely as first thought). F8 moredata still shows gap growing with more epochs (0.688→1.385 max), confirming the counter-hypothesis finding under the corrected methodology. F9 gap-vs-frequency curve reproduces F4's decreasing-gap-with-frequency shape using the corrected fully-offline per-context data. Negative control (nofork) shows a small (0.265 max), real, late-epoch3 gap uptick — verified against raw train.log numbers directly (not a script artifact); the existing data-driven verdict-writing logic in build_section_html() already reports this honestly rather than forcing a "clean negative control" narrative.
2026-07-25 — Frequency methodology fix + freeze ablation / more-data / gap-vs-freq (§11.11)
What/why: user review flagged that the G5 "truncation" was a symptom of a deeper bug — NGRAM_HIT_CUM_COUNT was full-training-run cumulative, so bin membership drifted over time (G6 documents the root cause: same probe token gets promoted into ever-higher bins as its count keeps growing). Fix: freeze the counter at the epoch1→epoch2 boundary; add ngram_raw_probe_log.jsonl (raw per-token loss + per-layer table/gate norms) so downstream analysis is decoupled from any pre-baked bin edges. User also confirmed (via AskUserQuestion) 3 new analyses this round: freeze ablation (n-gram tables+gates frozen after epoch1), longer-training-data (more shards), epoch3 gap-vs-frequency curve — deferring only the broader regime-bridge matrix expansion.
Runs: 5 H200 reruns on the patched staged train.py — first batch hit an epoch-indexing bug (G6: trained_epoch == 1 never fires since epoch starts at 1, not 0) that silently broke both the freeze-counter and the ablation switch despite clean completion; discovered by inspecting ngram_hit_count_frozen: [] in every summary, fixed (trained_epoch == 2), and all 5 rerun as "fixv2": nofork/rmsprop/muon_fixv2 (refresh of B.0/B.1/B.2 with the fixed capture), rmsprop_freeze_fixv2 (new, FREEZE_NGRAM_AND_GATE_AFTER_EPOCH1=1), rmsprop_moredata_fixv2 (new, 2 shards + MAX_TRAINING_STEPS 1000→2200). All verified complete to target step, freeze mechanism confirmed via both [ngram_hitbucket] froze hit counts... and [freeze_ngram_and_gate] step ...: froze N tensors... log lines at the correct epoch1/epoch2 boundary step.
Outputs: extract_norm_alignment.py RUNS dict repointed to the 5 fixv2 dirs (+2 new keys rmsprop_freeze/rmsprop_moredata); new functions load_epoch_freq_gap/log_bucket_gap_rows/build_gap_vs_freq_figure (streams the raw probe log, epoch==3 filter, log-spaced token-weighted bucketing) and build_pair_comparison_figure (gap-curve overlay, with an epoch-aligned x-axis option for moredata's different step budget); 3 new figures figB5_epoch3_gapvsfreq_trigram.svg, figB6_freeze_ablation_gap.svg, figB7_moredata_gap.svg; guide §11.10 refreshed with fixv2 data (numbers shifted <0.05 from pre-fix values, confirming the fix doesn't change prior qualitative conclusions) and new §11.11 (C.1/C.2/C.3).
Result: F7 (freeze ablation suppresses epoch2/3 gap to ~1/4-1/10 of baseline — n-gram table/gate updates during epoch2/3 are the primary gap driver), F8 (more training data does NOT shrink the gap — it grows monotonically across epochs, opposite of the original hypothesis; likely an artifact of "more epochs over the same doc set" rather than genuinely larger/more-diverse data — flagged as needing a follow-up experiment), F9 (epoch3 gap-vs-frequency curve independently reproduces F4's mid/low-frequency-driven gap shape using bin-edge-decoupled raw data).
2026-07-25 — §11.10 revision: negative-control setting + freqloss/interactive-panel fixes
What/why: user review of §11.10 flagged 3 issues: (1) manual.md's chain is actually 3 settings, not 2 — missing the + full n-gram, table AdamW, matrix AdamW (no target gap) negative control; (2) figB2_*_freqloss_*.svg lines truncate for low-frequency hit-buckets with no explanation (looked like broken data); (3) interactive per-token vs total-contribution views looked visually identical; (4) user also asked for the static twin-axis norm↔gap alignment SVG to become an interactive panel with bigram/trigram + norm-component + train/val/gap-loss toggles.
Runs: 3rd rerun on H200, reusing tmp/norm_verify_stage/ + remote work dir ngram_norm_verify_20260725 unchanged, env flipped to NANOGPT_NGRAM_OPTIMIZER=adamw NANOGPT_MATRIX_OPTIMIZER=adamw (G1 fix applied). Result dir 20260726_nano_adamw_adamw_111_full_tr1_1000step_seed42_normobs, completed cleanly to step 999, no crash.
Outputs: RUNS["nofork"] + RUN_ORDER added to extract_norm_alignment.py; build_freqloss_figure/svg_line_chart now draw an end-marker circle + footnote when a bin's curve truncates before the chart's xhi (G5); new build_alignment_interactive_html → figB4_<key>_alignment_interactive.html (bigram/trigram × gate_rms/raw_value_rms/gated_value_rms × train/val/gap overlay), old figB3_* kept as static thumbnail; INTERACTIVE_TEMPLATE per-token/contrib toggle now shows an explicit units note and y-range readout per view; guide §11.10 regenerated with B.0/B.1/B.2 sub-blocks (B.0's verdict text is data-driven, not copied from B.1/B.2).
Result: F6 (negative control shows no sustained gap onset, max per-epoch gap ≈0.38 vs final gap 0.6/3.2 for RMSProp/Muon) — confirms gap requires an optimizer asymmetry (table RMSProp or matrix Muon), not merely the presence of n-gram tables.
2026-07-25 — manual.md task #2: RMSProp/Muon norm↔gap alignment
What/why: extend §11.9 time-lag verification to two nanogpt_original settings (table RMSProp; + Muon). Collect gate/table/gated norms + context-frequency-bucketed loss; test whether norm growth aligns with gap onset.
Runs: 2 reruns on H200 (see §Runs). First attempt hit G1 (config leak) → train stuck 7.2; fixed (layernorm/tied) → reproduced originals within ±0.12.
Outputs: figB* SVGs + interactive HTML + figB_alignment.json; guide §11.10 + TOC; patched staged train.py w/ NGRAM_HIT_BUCKET_CAPTURE; tool extract_norm_alignment.py.
Result: F2 (RMSProp: table-content jump aligned w/ e3 fork), F3 (Muon: norm builds e1-2, gap peaks e3 → time-lag), F4 (mid-freq buckets drive gap). Both reruns reproduce known regimes.