Episode Details

Crossposted from Belief Updates, the Simplex blog, where several of the figures are interactive. By Kyle J. Ray, Paul M. Riechers, and Adam S. Shai (Simplex, Astera Institute).

Telescoping cones recovered by linear regression from a transformer's residual-stream activations. Each component grows and shrinks in accordance with in-context evidence.

Introduction

Perhaps the defining feature of LLM pretraining data is its heterogeneity. The training corpus spans not only the collected and varied textual works output by the whole of humanity, but also those generated by machines, data collection devices, and more. Such a large and varied corpus is often appealed to as an explanation for the abilities of modern LLMs. But the statistical structure of data created by a diverse set of generators also implies a particular computational structure for the next token prediction task, and, as we will see, for the geometric arrangement of the internal activations in LLMs.

In order to understand the structure of the next token prediction task over data generated from many different sources, and its implications for the geometric structure of activations in neural networks, we will:

• Start by introducing the concept of nonergodicity, which is an important [...]

---

Outline:

(00:47) Introduction

(03:25) LLM Training Data is Nonergodic

(04:59) Two coins: the simplest example of a nonergodic process

(09:33) Nonergodic Generators of Data and the Task of Prediction over them

(10:36) HMMs as Latent Generators of Token Sequences

(11:43) The Task of Prediction and Belief State Geometry

(13:17) Nonergodicity, Prediction, and Telescoping Geometry!

(13:52) Nonergodic Composition

(16:02) Belief Geometry over Nonergodic Data

(19:32) Does this geometry show up in trained models?

(23:54) Did it have to be this way?

(25:56) Parting thoughts

(29:47) Appendix

(29:50) Acknowledgments

(30:56) The Mess3 process

(31:22) Training details

(33:25) Generators of Data and the Geometry of Beliefs

(35:03) HMMs and their transition operators

(37:45) Prediction Over Data Generated by HMMs

(41:54) The geometry of beliefs

(43:20) Citation

(43:33) References

The original text contained 13 footnotes which were omitted from this narration.

---

First published:
September 10th, 2026

Source:
https://www.lesswrong.com/posts/JfJ4WTRHmooPBWRFv/the-geometry-of-nonergodic-composition

---

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Listen Now