知识体系的价格

Under tight step budgets (T <= ~500 steps) where candidates share identical architecture and loss function, optimization progress is dominated by step magnitude; the largest learning rate that maintains numerical stability (e.g., adaptive optimizer lr ~ 1e-3 on small MLPs) achieves superior convergence compared to smaller learning rates (lr <= 1e-4) that leave weights under-trained near initialization.

操作化：同架构、T≤512、两方都是自适应优化器：小的一方 lr≤1e-4、大的一方 lr≥3e-4 且至少 3 倍时，lr 大者胜。原文按名义学习率比较，不换算 Δ。

Learning non-linear feature interactions (such as XOR) requires moving weights significantly beyond the near-linear initialization regime; under short optimization budgets (T <= ~256 steps), undersized learning rates (lr <= 1e-4) produce insufficient logit displacement, leaving classification loss stuck near chance level (cross-entropy ~ ln(2) ≈ 0.693).

操作化：xor、T=256、同架构：一方 lr≤1e-4、另一方 lr>1e-4 时，后者胜（按名义 lr，不分优化器）。

When fitting continuous MSE regression targets under short step budgets (T <= ~500 steps), Adam at standard learning rates (lr ~ 1e-3) achieves substantially lower test MSE than unaccelerated SGD at the same nominal learning rate, because Adam's per-parameter gradient normalization maintains effective parameter updates of scale ~lr even after the output head aligns near the target mean and raw MSE gradients become small.

操作化：MSE 任务、T≤512、同架构：Adam/AdamW 对无动量 SGD，两者名义 lr 相同时 Adam 胜。

Under fixed short step budgets (T ~ 500-2000 steps) with identical architectures, candidate ranking across optimizers is determined by cumulative parameter displacement: Adam/AdamW produces sign-normalized updates yielding total displacement ~lr * T, Adagrad bounds total displacement to ~2 * lr * sqrt(T) due to step decay, and unaccelerated SGD at small learning rates on small MSE gradients produces negligible displacement (~lr * T * ||grad||); optimizers achieving larger stable cumulative displacement (e.g., AdamW lr ~ 1e-4 yielding ~0.2 displacement over 2000 steps) achieve superior convergence compared to displacement-limited optimizers.

操作化：同架构、T∈{512,1024,2048}；只覆盖 Adam/AdamW、Adagrad 和不带动量的 SGD（原文没提 RMSprop 和动量 SGD）；Δ 大者胜。

Under tight step budgets (T <= ~256 steps) on language modeling tasks over vocabulary size V, Adam or AdamW at standard learning rates (lr ~ 3e-4 to 1e-3) achieves substantially greater cross-entropy reduction than SGD with momentum or small-learning-rate optimizers (lr <= ~3e-5), because per-parameter gradient normalization maintains effective parameter updates of scale ~lr regardless of gradient magnitude, driving loss well below the uniform entropy baseline ln(V).

操作化：bigram、T=256、同架构：Adam/AdamW 且 lr∈[3e-4,1e-3]，胜动量 SGD 或 lr≤3e-5 者。

Under short optimization budgets (T <= ~256 steps), the choice of optimizer and effective learning rate dominates architectural differences: Adam or AdamW at standard learning rates (lr ~ 1e-3) achieves roughly 10-30x faster cross-entropy reduction than reduced learning rates (lr <= ~1e-4), whereas severely undersized learning rates (e.g., RMSprop at lr <= ~3e-5) leave output logits near initialization and test cross-entropy stuck near chance level (ln(2) ≈ 0.693).

操作化：T=256、交叉熵任务，任何配方不同的对（含架构不同）：Adam/AdamW 且 lr≥1e-3 胜 lr≤1e-4 者；RMSprop 且 lr≤3e-5 输给 lr≥1e-4 的符号归一优化器。

On classification tasks with zero linear baseline signal (e.g., XOR-like problems), initial cross-entropy loss remains stuck near chance baseline (ln 2 ~= 0.693) until parameters undergo sufficient cumulative displacement away from initialization to form non-linear feature interactions; under short step budgets (T <= ~256 steps), candidate progress is strictly ordered by stable effective step size (e.g., Adam lr = 1e-4 vs 3e-5).

操作化：xor、T=256、同架构的不同配方：Δ 大者胜。

On complex multivariate regression tasks with non-linear multiplicative and oscillatory targets under moderate step budgets (T ~ 2000 steps), AdamW at standard learning rates (lr ~ 3e-3) achieves cumulative parameter displacement ~lr * T (~6.0), enabling sufficient weight exploration to fit complex non-linear structures, whereas severely reduced learning rates (lr <= 3e-5) restrict displacement to ~0.06, leaving parameters trapped near initialization; intermediate adaptive optimizers (e.g., RMSprop at lr ~ 1e-4) yield intermediate displacement but are limited over long horizons by step-decay scaling relative to sign-normalized updates.

操作化：多变量回归、T=2048、同架构：Adam/AdamW 且 lr≥1e-3，胜 lr≤3e-5 者或 lr≤1e-4 的 RMSprop。

On classification tasks with zero linear baseline signal (such as XOR), raw output gradients near chance initialization are near zero, causing unaccelerated SGD or SGD with momentum at moderate learning rates (lr <= ~1e-3) to stall on the chance loss plateau (CE ≈ ln 2); adaptive optimizers (Adam/RMSprop) maintain per-parameter effective step sizes of scale ~lr regardless of raw gradient magnitude, driving cumulative parameter displacement sufficient to break symmetry and form multiplicative feature interactions.

操作化：xor 任务、同架构：Adam/AdamW/RMSprop 对 lr≤1e-3 的 SGD（含动量），自适应胜。

Under short step budgets (T <= ~256 steps), Adagrad initialized with zero accumulators yields cumulative per-parameter displacement of ~2 * lr * sqrt(T) (achieving ~0.03 at lr ~ 1e-3), outperforming tiny-learning-rate RMSprop or Adam (lr <= ~3e-5, cumulative displacement ~lr * T <= ~0.008) by providing ~4x greater total parameter movement to escape near-initialization loss plateaus.

操作化：T=256、同架构：Adagrad 且 lr≥1e-3，胜 lr≤3e-5 的 Adam/AdamW/RMSprop。

For highly curved decision boundaries (such as 2-D spiral classification) with unnormalized inputs under short step budgets (T <= ~500 steps), unaccelerated or low-learning-rate momentum SGD (effective lr <= ~3e-4) stalls near the chance plateau where raw gradients are small, whereas sign-normalized adaptive optimizers (Adam, RMSprop) generate cumulative parameter displacement (~lr * T) independent of raw gradient scale to reshape the decision boundary.

操作化：spiral、T≤512、同架构：Adam/AdamW/RMSprop 胜有效学习率 lr/(1−m)≤3e-4 的 SGD。

When fitting complex non-linear decision boundaries under short step budgets (T <= ~256 steps), candidate ranking across adaptive optimizers is governed by cumulative parameter displacement (~lr * T for Adam vs ~2 * lr * sqrt(T) for Adagrad); higher adaptive learning rates (lr ~ 3e-3 vs 1e-4) and reduced second-moment decay (beta2 ~ 0.95) maximize early parameter displacement to escape the chance classification plateau.

操作化：分类任务（xor/spiral/tabular）、T=256、同架构、两方都是自适应优化器：Δ 大者胜；Δ 相同时 β2=0.95 者胜。

On classification tasks with zero linear baseline signal (such as XOR) under short step budgets (T <= ~256 steps), sign-normalizing adaptive optimizers (Adam or AdamW) at moderate learning rates (lr ~ 3e-4) achieve cumulative parameter displacement (~lr * T ≈ 0.077) nearly two orders of magnitude larger than Adagrad at severely undersized learning rates (lr <= ~3e-5, displacement ~ 2 * lr * sqrt(T) ≈ 0.001), providing sufficient parameter movement to escape the chance loss plateau (CE ≈ 0.693) and form multiplicative feature interactions.

操作化：xor、T=256、同架构：Adam/AdamW 且 lr≥3e-4，胜 lr≤3e-5 的 Adagrad。

Optimizer comparison,SAME model/data/budget: use Delta as a progress proxy ONLY below task/model saturation. T256 can be32epochs; neither step count nor nominal parity proves adaptive or max-Delta wins. Adam/AdamW/RMSprop:lr*T;Adagrad:2*lr*sqrt(T);SGD:lr*T*|g|/(1-m),g~.01 (bigram.003,momentumMSE.03,K1019). Check K1004/K1071-K1073 and target/batch first;close mixed-recipe estimates:K1012. Old largerDelta~85% backtests were ceiling-filtered,not a law. Byte-identical q_561d81/q_21a335 recipes reverse winners on different targets/splits. Long XOR optimizer reversal K1119 is NOT a low-rate rule;no universal ceiling.

操作化：同架构、同预算、T∈[256,2048] 的不同配方对，按 Δ 取大。Δ：Adam/AdamW/RMSprop 为 lr·T，Adagrad 为 2·lr·√T，SGD 为 lr·T·g/(1−m)，g 取原文默认 0.01。原文的“只在饱和以下”没有给出可执行阈值，单独使用时不做饱和过滤；原文括注的分族校准（bigram .003、动量 MSE .03）引自 K1019，由 K1019 这条规则提供。

Multivariate plain MSE,SAME model/data/budget: Adam/RMS rate ratio>=3 is NOT sufficient. Plain d5w192 GELU/LN11001,wd0,b16,Adam(.9,.999),RMS.0003: on nested0eb811 Adam.003 loses atT256 by+.0303874 MSE,paired95%CI[.0102102,.0498590],2/10Adam seedwins;T1024 gap+.0817989 butCI crosses0. Same deep polynomial5a829e: equal-rate Adam.0003 beatsRMS atT256/1024 by.00863518/.0121065,bothCIs<0. Shallow resd1w16 GELU/zeroLN,T256/b16:equal-rateRMS.0003 wins4/4targets(K1187). Old ratio>=3 history32/32(19questions),lab16/16(shallowT256) did NOT transfer. Rb5002830bf:16fresh1024paths/160seeds/fail0;T256 prefixes,not reruns. K1114's SAME-AdamW caution cannot rank cross-optimizer pairs.

操作化：多变量回归、MLP、T∈[256,1024]，Adam/AdamW 对 RMSprop 的同架构对：禁止按 Δ 下结论。

Mixed-optimizer/architecture rankings: close Delta estimates are NOT winner evidence; even a25% edge is inconclusive. Below task/model saturation,old >=4x gap favors progress35/40;within3x architecture wins11/19,only a weak preference. Once both Delta>=.3,extra wins~62%;check task/model/budget(K1004/K1063/K1071-K1073). Delta<.005 does NOT establish tied CE=lnV/ln2 or MSE=Var(y): initialization logit scale and target mean affect baseline loss. Do not turn weak architecture tendencies into a complete rank there. Nominal parity supplies no automatic adaptive-over-SGD tie-break(q_826eab),nor RMSprop-over-Adagrad law.

操作化：K1012 拆出的第一条主张。只管架构和配方都不同的对；小的一方 Δ<0.3（原文“两者都 ≥.3 后只有 62%”）且两者 Δ（K1019 校准）相差 ≥4 倍时，Δ 大者胜（原文 35/40）。

Mixed-optimizer/architecture rankings: close Delta estimates are NOT winner evidence; even a25% edge is inconclusive. Below task/model saturation,old >=4x gap favors progress35/40;within3x architecture wins11/19,only a weak preference. Once both Delta>=.3,extra wins~62%;check task/model/budget(K1004/K1063/K1071-K1073). Delta<.005 does NOT establish tied CE=lnV/ln2 or MSE=Var(y): initialization logit scale and target mean affect baseline loss. Do not turn weak architecture tendencies into a complete rank there. Nominal parity supplies no automatic adaptive-over-SGD tie-break(q_826eab),nor RMSprop-over-Adagrad law.

操作化：K1012 拆出的第二条主张。配方不同的任何对，两者 Δ（K1019 校准）相差不到 25% 时禁止下结论。

SGD vs ADAPTIVE optimizers: family-calibrated Delta is a proxy,NOT a plain-vs-momentum SGD rule. Fit368pairs. MSE momentum.9 raw|g|~.03: Adam-equivalent lr/3,largerDelta correct59/71 vs54/71 at .01; wins11/19 when adaptiveDelta<=4x its .01-estimate. Classification131/146 and plainSGD MSE keep .01;do NOT infer a momentum bonus at equal lr/(1-m). Bigram |g|~.003:m.9 ~Adam lr/30,plain~lr/300;correct68/70 vs63/70 at .01. Momentum LM beats adaptive1/18,nominal parity1/4. PlainSGD wins3/141 when adaptiveDelta>15x larger. MLP calibration assumes>=1LN;zeroLN:K1038. Parity needs direct matched evidence(R838be7703a),NOT this multiplier.

操作化：只管恰好一方是 SGD 的同架构对；MLP 须至少一层 LayerNorm（原文：零 LN 见 K1038）。g：bigram 0.003，MSE 且带动量 0.03，其余 0.01；Δ 大者胜。

Bigram LM,SAME model/data only: neither lower RMSprop rate nor Delta .1-.3 is a safety rule. Do NOT penalize RMSprop across changed widths/depth/FF or rank recipes by nearest-.2. GRU64x1,V24/L24/alpha.8,T256/b16: Adam/RMS.003 beat AdamW.0003(q_53593d);longer GRU reversals onlyK1152's exact rates. Legacy d32/d64 band:>=2x extraDelta loses TF43/44,GRU5/6(CE+2-20%);below band higher wins13/13,14/15. d128 .03-.1 rivals beat larger11/11,not a universal width penalty; conservative<.03 need not freeze(K1102). q_409769 heads2/FF64/wd.001/T512 RMS.003 wins outside older controls(Rabae8971ee). Nearest-.2 52/65 is confounded;data/init/budget matter.

操作化：bigram 同模型对。d_model≤64：小的一方 Δ 在 0.1–0.3 带内、大的一方 ≥2 倍时，Δ 小者胜；两者都 <0.1 时 Δ 大者胜。d_model=128：小的一方在 0.03–0.1、大的一方 >0.1 时，Δ 小者胜。Δ 用 K1019 校准。

Univariate MSE,SAME depth>=2 width>=64 MLP: Delta>=1 is a relative overshoot RISK,not automatic loss to Delta.1-.5. Older high wins:w64 0/12(MSE x8.8),w128 5/18(x5.5),w256 1/13(x4.4);controlled moderate wins11/12. Counterexample K1115:zeroLN d2/w128 LeakyReLU Adam,T2048,Delta6.144 MSE.01578 beats Delta.2048 MSE.35314;q_af3ba4 also favors high over most slow recipes. No universal zeroLN exemption or near-frozen rival cutoff. Width<=32,depth1 and multivariate are outside this scope,NOT high-rate safety evidence(multivariate K1114/K1117). Check target,optimizer,budget and trained rival.

操作化：单变量回归、MLP depth 2–5、width 64–256、两方都是自适应优化器：大的一方 Δ≥1、小的一方 Δ 在 0.1–0.5 时，取小者（原文历史上高 Δ 只赢 6/43，受控实验适中赢 11/12）。原文说这是“风险，不是自动输”，也可读成只禁止下结论。

When fitting continuous targets with a large constant offset |E[y]| >> output_init, SGD with momentum m scales effective learning rate to lr / (1 - m) with a bias adaptation time constant of ~1 / (2 * lr / (1 - m)) steps; under tight step budgets (T ~ 256), this enables rapid early bias alignment and MSE reduction compared to unaccelerated optimizers.

操作化：MSE 任务、T=256、同架构的两个 SGD，lr 相同、动量不同：动量大者胜。

Under tight step budgets (T <= ~500 steps) at small Adam/AdamW learning rates (lr <= ~1e-4), lowering the second-moment decay factor from beta2 = 0.999 to beta2 = 0.95 accelerates convergence by enabling second-moment variance tracking to adapt within fewer steps (~1 / (1 - beta2)), providing faster effective step-size calibration and superior early loss reduction.

操作化：T≤512、同架构 Adam/AdamW 对，两者 lr 相同且 ≤1e-4、β2 不同：β2=0.95 者胜（其他不变）。

Under short step budgets (T <= ~256 steps) on continuous multivariate regression tasks, Adam at standard learning rates (lr ~ 3e-4) achieves cumulative parameter displacement of ~lr * T (~0.08), enabling substantial fitting of non-linear target interactions, whereas Adam at severely reduced learning rates (lr <= 3e-5) restricts displacement to ~0.008, leaving parameters trapped near initialization with minimal capacity to fit complex non-linear structure.

操作化：多变量回归、T=256、同架构的 Adam/AdamW 对：lr≥3e-4 胜 lr≤3e-5。

On bigram next-token prediction tasks using transformer language models under short step budgets (T <= ~256 steps), candidate progress is governed by parameter displacement speed in token embeddings and output heads; sign-normalized adaptive optimizers (Adam/AdamW at lr ~ 3e-4) achieve cumulative parameter displacement of ~lr * T, outperforming Adagrad (whose displacement decays as ~2 * lr * sqrt(T)) and momentum SGD (whose updates scale with raw cross-entropy gradients that shrink once logits match the unigram marginal distribution).

操作化：bigram、Transformer、T=256、同架构：Adam/AdamW 且 lr≥3e-4，胜 Adagrad 或动量 SGD。

On classification tasks with zero linear baseline signal (such as XOR), Adam or AdamW at standard learning rates (lr ~ 1e-3) achieves cumulative parameter displacement of ~lr * T (~2.0 over T ~ 2000 steps) sufficient to fully escape the chance loss plateau and drive test cross-entropy well below 0.2, whereas reduced learning rates (lr ~ 3e-4) restrict cumulative displacement to ~0.6, leaving the model mid-transition at the budget limit.

操作化：xor、T=2048、同架构 Adam/AdamW 对：lr 大者胜。

The architecture-selection regime boundary is at critical displacement D* ≈ 0.05: for D < D* the per-layer gradient-preservation multiplier dominates and each LayerNorm layer contributes effective displacement gain ×1.4 (so k LayerNorm layers ≈ 1.4^k boost), making normalization coverage the primary differentiator; for D > D* representational capacity dominates and the ranking inverts to depth ≥ width > residual > LayerNorm-coverage, with the transition sharp within ±0.01 of D*.

操作化：同优化器、不同 MLP 架构：由共享配方算 Δ（K0019/K1001 公式，SGD 取 g=0.01）。Δ<0.05 时 LayerNorm 层数多者胜；Δ>0.05 时按 depth、width、residual、LN 层数依次取大。Δ 由位移规则提供，这是一条前提链。

Fitting complex non-linear decision boundaries (such as a 2-turn spiral) under short step budgets (T <= ~500 steps) requires deep residual architectures (depth 4-6 with LayerNorm) paired with adaptive optimizers at standard learning rates (lr ~ 1e-3); underpowered optimizers or shallow unnormalized networks remain trapped near initialization at chance binary cross-entropy (CE ≈ ln(2) ≈ 0.693), whereas adaptive deep residual MLPs achieve substantially lower test loss (CE < ~0.5).

操作化：spiral、T≤512、同优化器且为自适应、lr≥1e-3：depth 4–6 且 residual 且有 LN 者，胜 depth≤2 且无 LN 者。

On complex 2-D classification tasks with high-frequency decision boundaries (e.g., multi-turn spirals), deep normalized residual MLPs (>=3 residual blocks with LayerNorm) achieve higher decision-boundary curvature and faster cross-entropy reduction per step than shallow MLPs (1-2 hidden layers) of comparable width, which require substantially larger width and longer step budgets to resolve intricate boundary turns.

操作化：spiral、同优化器：depth≥3 且 residual 且有 LN 者，胜 depth≤2 且宽度相差不超过 2 倍者。

For first-order lookup tasks such as bigram next-token prediction under short step budgets (T <= ~256 steps), shallow single-layer transformer architectures trained at standard learning rates (lr ~ 1e-3) achieve faster cross-entropy reduction than deeper multi-layer stacks (e.g., 4-layer post-LayerNorm), because single-layer models directly map token embeddings to output logits without optimization overhead or backpropagation attenuation through unnecessary hidden layers.

操作化：bigram、T=256、同优化器、lr∈[3e-4,3e-3]：1 层 Transformer 胜 ≥2 层 Transformer。

On bigram language modeling tasks where the target distribution depends strictly on immediate preceding tokens P(y_t | x_{t-1}), recurrent architectures (such as 2-layer GRUs) achieve superior sample efficiency and faster convergence under short step budgets (T <= ~512 steps) compared to transformer architectures, because recurrent hidden state propagation directly captures single-step temporal lookup without requiring learned positional encodings or attention patterns.

操作化：bigram、T≤512、同优化器：GRU 胜 Transformer。

For tabular classification under short step budgets (T <= ~256 steps), moderate-depth residual architectures (3-4 hidden layers) achieve faster loss reduction than deeper non-residual or over-deep stacks (depth >= 5 blocks) because residual shortcut paths allow early shallow-like signal propagation while preserving nonlinear expressive capacity.

操作化：tabular、T=256、同优化器：residual 且 depth 3–4 者，胜 depth≥5 者。

At small learning rates (lr <= ~3e-5) and tight step budgets (T <= ~512 steps), non-residual MLPs with LayerNorm on all hidden layers achieve faster convergence than partially normalized configurations, because maintaining unit-RMS activations across all layers prevents cumulative forward signal attenuation (~0.4-0.5x per unnormalized layer) and maximizes network output sensitivity per unit parameter displacement.

操作化：T≤512、同优化器且 lr≤3e-5：非残差且每层都有 LN 者，胜部分层有 LN 者。

In pre-activation LayerNorm residual MLPs trained with small learning rates (lr <= ~1e-4) under short step budgets (T <= ~256 steps), the dominant identity path keeps early predictions near a linear mapping until residual branch weights grow, slowing early logit displacement and compounding optimization latency compared to non-residual or unnormalized architectures.

操作化：T=256、同优化器且 lr≤1e-4：非残差或无 LN 者，胜 residual 且有 LN 者。

For shallow, narrow MLPs (depth 2-3, width <= 32) on non-linear binary classification with piecewise feature interactions under short step budgets (T <= ~256 steps) trained with Adam at standard learning rates (lr ~ 1e-3), architectures combining residual connections and LayerNorm achieve lower test cross-entropy than plain feedforward architectures of identical depth and width, because LayerNorm stabilizes pre-activation distributions against gradient saturation while residual paths enable incremental feature updates to escape near-chance loss plateaus.

操作化：tabular/xor、T=256、同优化器 Adam/AdamW 且 lr∈[3e-4,3e-3]、depth 2–3、width≤32 且两方深宽相同：residual 且有 LN 者胜无残差无 LN 者。

On non-linear univariate regression tasks under short step budgets (T <= ~256 steps) with unaccelerated SGD at moderate learning rates (lr ~ 3e-3), residual connections enable deeper MLPs (depth >= 4) to achieve lower test MSE than shallower non-residual MLPs (depth 2-3), because identity shortcuts preserve direct gradient pathways (~lr * T * ||grad||) across all layers, preventing the compounding gradient attenuation through sequential weight matrices that restricts early-layer parameter displacement in non-residual networks.

操作化：单变量回归、T=256、同优化器为无动量 SGD 且 lr≥1e-3：residual 且 depth≥4 者，胜非残差 depth 2–3 者。