Episode Details

What does it really mean to understand how something learns, whether it's a child, a brain, or a neural network? In today’s episode, we sit down with Prof. Andrew Saxe, Professor of Theoretical Neuroscience & Machine Learning at University College London, to explore how mathematics can help us understand learning processes in all kinds of networks. We dive into the two-way street between how and what we learn, examining the way learnable content and the learning process shape each other. Our conversation anchors itself in Prof. Saxe’s early projects on language development in infants, current interests in knowledge generalization, and future aims for experimentalist-theorist collaborations, and we can’t wait for you to give it a listen!

In this episode, we cover:

(1:13) Motivations behind studying the learning process

(5:24) When did neuroscience and psychology come into the picture?

(8:06) Projects and people that defined his trajectory

(12:33) Simple mathematical models give rise to complex dynamics. Why focus on simplicity?

(15:16) Current and future focus points of the lab? (in the era of LLMs…)

(20:27) Sparse reward problems in learning processes

(21:46) How hierarchy helps us learn

(24:12) How our experience shapes our language structure

(29:02) Lab’s Three Axes of Learning

(34: 35) Invariant manifolds and network initialization schemes

(36:50) The manifold hypothesis

(39:50): Cross talk between theorist and experimentalist

(41:26) The NeuroAI research program

Further reading (papers, books and more)

• Read more about Prof. Andrew Saxe, his Theory of Learning Lab and his 2025 Blavatnik Finalist Award
• A mathematical theory of semantic development in deep neural networks (2019)
• What makes language learnable? Dive into Noam Chomsky’s work to learn more.
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model (2023)
• The Story of Your Life (mistakenly called “The Story of You” during the episode)
• Simon Kirby and how our learning capabilities are crucial for structuring our language
• Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures (20250)
• Scaling Learning Towards AI (introduces LeCun and Bengio’s AI Set)

Episode host and producer: Sabrina Du

Contact: igem@mcgill.ca