Episode Details

Artificial intelligence works by routing data through layered mathematical functions whose parameters get tuned through calculus based optimization, a process that used 415 terawatt hours of electricity worldwide in 2024. Every input, whether a sentence of text, a photograph, or a sensor reading, is first converted into a list of numbers. Those numbers pass through many layers of arithmetic, each layer shaped by parameters learned from data rather than programmed by hand. A training procedure repeatedly compares the system’s output to a known correct answer and adjusts every parameter using a calculus technique called backpropagation, in much the same way an engineer uses adjoint sensitivity analysis to trace how a design change ripples backward through a coupled physical model. Once training stops, the same fixed arithmetic runs forward only, at a fraction of the original computational cost, to produce predictions on data it has never encountered.

What Mathematical Operation Actually Happens Inside a Neural Network?

A single artificial neuron performs one operation: it multiplies each incoming number by a weight, adds those products together with a bias term, and passes the sum through a nonlinear function. The nonlinear step matters. Without it, stacking layers would collapse into a single linear transformation, no more expressive than one matrix multiplication no matter how many layers were added. With a nonlinearity such as the rectified linear unit, which simply outputs zero for any negative input and passes positive input through unchanged, a network of stacked layers can approximate almost any continuous function given enough neurons, a result formalized in the universal approximation theorems of the late 1980s and early 1990s.

For an engineer trained in linear systems, the closest analogy is a block diagram in which each block applies a gain and then a saturation nonlinearity, similar to the transfer function of an amplifier stage that clips at its rails. A neural network layer is that block diagram generalized to thousands of inputs and thousands of outputs simultaneously, represented as a matrix multiplication followed by an elementwise nonlinear function, and a full network is dozens or hundreds of these blocks connected in series. The parameters inside those matrices, the weights and biases, are the only quantities the network actually learns. A modest network with three layers of one thousand neurons each already contains roughly two million such weighted connections. A large language model released in 2020 disclosed 175 billion parameters, and systems built three years later used training computation on the order of one hundred times greater still.

Modern networks also commonly include a residual connection, a direct path that adds a layer’s input back to its output alongside the normal calculation, introduced in the mid 2010s to solve a practical training problem: without that direct path, gradients computed during backpropagation tend to shrink toward zero as they travel backward through many layers, leaving the earliest layers of a deep network barely updated at all. A residual connection functions much like a feedback path around a control block: it guarantees a signal can pass through a stage largely unmodified when the learned transformation at that stage is not yet useful, which is part of why networks with over one hundred layers became trainable at all.

Nothing about this arithmetic encodes physical law, chemical kinetics, or a governing differential equation. The network has no built in notion of conservation of mass, momentum, or energy unless a designer explicitly adds a penalty term to enforce one. It is, at its core, a very large adjustable curve fit, and every claim about what the system understands has to be evaluated against that fact.

How Does a Neural Network Learn From Data?

Training begins by setting every weight to a small random number. The netwo