Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

Joshua Shay Kricheli1‖, Alexander Lawrence Reid1‖, Soumajyoti Sarkar2*, Venkata Gandikota1, Paulo Shakarian1
1Syracuse University 2Amazon AGI Foundations
‖Equal contribution. *This work does not relate to the author's position at Amazon.

NeurIPS 2026 · Sydney, Australia

NeurIPS Syracuse University Syracuse University Institute for Artificial Intelligence Leibniz Lab

Summary

Abstract

Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$. Following Chinchilla-style compute-optimal training, many studies fit scaling laws from runs at one or a few fixed tokens-per-parameter (TPP) ratios $k$, with $D = kN$. We show that this collinear design, combined with the empirically common near-equality of the exponents governing $N$ and $D$, induces an inherent ill-conditioning in the Gauss-Newton least-squares problem: the condition number of the design grows as the inverse square of the gap between the $N$- and $D$-exponents. The scale coefficients become practically unidentifiable, with confidence intervals inflating by orders of magnitude, yielding a "sloppy" model whose extrapolations degrade sharply off the training ray. We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that characterizes well-conditioned estimation to leading order. Empirically, in controlled experiments with 5M-77M-parameter models, non-collinear designs outperform collinear ones on held-out data with a 97.3% win rate across four laws and five corpora, and 98.7% in bfloat16. We further show the degeneracy is rooted in Jacobian geometry and is not an artifact of the loss function: any smooth objective whose Gauss-Newton weights are bounded above and away from zero inherits the same ill-conditioning.

One ray cannot separate model and data

The Chinchilla rule trains at about 20 tokens per parameter, $k = D/N \approx 20$ (Hoffmann et al., 2022). On a ray $D = kN$ the Chinchilla law (Hoffmann et al., 2022) $L = AN^{-\alpha} + BD^{-\beta} + E$ becomes

$$L(N, kN) = A N^{-\alpha} + B k^{-\beta} N^{-\beta} + E,$$

so only the exponent gap $\varepsilon = |\alpha - \beta|$ separates $A$ from $B$. The scale pair's condition number grows as $\kappa_{A,B} = \Theta(\varepsilon^{-2})$, and the confidence interval on $A$ is $\Omega(\varepsilon^{-1})$ times the one on the identified combination $\psi = A + Bk^{-\alpha}$.

A threshold checkable before training

For a collinear design with ratios $k_1 < \cdots < k_K$, to leading order $\kappa_{A,B} \leq \kappa_{\mathrm{target}}$ whenever

$$V_K = \frac{1}{K}\sum_{\ell=1}^{K} k_\ell^{-2\beta_{\mathrm{eff}}} - \Bigl(\frac{1}{K}\sum_{\ell=1}^{K} k_\ell^{-\beta_{\mathrm{eff}}}\Bigr)^{2} \;\geq\; \tau_K = \frac{\bigl(K + \sum_{\ell} k_\ell^{-2\beta_{\mathrm{eff}}}\bigr)^{2}}{K^{2}\,\kappa_{\mathrm{target}}},$$

and $\kappa_{A,B} \leq \kappa_{\mathrm{target}}$ exactly when $V_K \geq \tau_K/(1 + 1/\kappa_{\mathrm{target}})^2$. One ray never passes; two rays pass once their spread $k_2/k_1$ reaches $R_{\min}(k_1)$.

Results

Comparison (NC vs. CO)NC wins
Full fan, FP32 (1,460 of 1,500)97.3%
Full fan, BF16 (296 of 300)98.7%
Budget-matched below the threshold, Chinchilla (Hoffmann et al., 2022)68.5%
Budget-matched below the threshold, Droppo-Elibol (Droppo and Elibol, 2021)57.4%
Budget-matched below the threshold, Kaplan (Kaplan et al., 2020)80.6%

Four laws: Chinchilla (Hoffmann et al., 2022), Kaplan (Kaplan et al., 2020), Droppo-Elibol (Droppo and Elibol, 2021) and repeated-data (Muennighoff et al., 2025).

Fitted loss surface with holdout points
Fitted surface with holdout points.
Holdout R squared against TPP coverage
Holdout $R^2$ against TPP coverage.
Induced isoFLOP curves
Induced isoFLOP curves.

Kaplan law (Kaplan et al., 2020) on RedPajama, first epoch; CO in blue, NC in red.

Design recipe

  1. Pick the lower ratio $k_1$ and read the minimum spread $R_{\min}(k_1)$ at $\kappa_{\mathrm{target}} = 100$.
  2. Train on two ratios at the ends of that spread, runs split evenly, log-uniform in $N$.
  3. Check the exact $\kappa_{A,B}$ of the planned $(N, D)$ before training.

References

  1. J. Hoffmann et al. An empirical analysis of compute-optimal large language model training. NeurIPS 2022. arXiv:2203.15556
  2. J. Kaplan et al. Scaling laws for neural language models. arXiv:2001.08361, 2020.
  3. J. Droppo and O. Elibol. Scaling laws for acoustic models. Interspeech 2021. doi:10.21437/Interspeech.2021-1644
  4. N. Muennighoff et al. Scaling data-constrained language models. JMLR 26(53), 2025. jmlr.org
  5. M. K. Transtrum, B. B. Machta and J. P. Sethna. Why are nonlinear fits to data so challenging? Phys. Rev. Lett. 104, 2010. doi:10.1103/PhysRevLett.104.060201
  6. M. K. Transtrum et al. Perspective: Sloppiness and emergent theories in physics, biology, and beyond. J. Chem. Phys. 143, 2015. doi:10.1063/1.4923066

BibTeX

@inproceedings{kricheli2026tpp,
  title     = {Tokens-per-Parameter Coverage Is Critical for Robust {LLM} Scaling Law Extrapolation},
  author    = {Kricheli, Joshua Shay and Reid, Alexander Lawrence and Sarkar, Soumajyoti and Gandikota, Venkata and Shakarian, Paulo},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  note      = {arXiv:2605.08541}
}