Reliability Scales Inversely: Bigger Language Models Compound Mistakes Faster

תקציר מקורי באנגליתarXiv:2607.18292v2 Announce Type: replace Abstract: As language models scale, answers start truer but degrade faster: scaling buys capability but erodes reliability. The knowledge-gap account -- more data, retrieval, or scale -- misses an auto-regressive risk residual that increases with scale: the model commits to a low-probability token, conditions on it as established, and snowballs. We track this through per-position disagreement $\delta = \log p_M - \log p_O$ against a stronger same-family oracle, whose second moment splits exactly into bias$^2$ $\mathrm{KL}(p_M \,\|\, p_O)^2$ and risk $\mathrm{Var}[\delta]$. Across three model families, we present four findings: (i) under scaling, the knowledge gap falls up to $7\times$ while knowledge degradation grows up to $39\times$; (ii) at a fa