A Neural Network Is Not A Brain

A neural network is not a model of the brain. It is a mathematical function with millions or billions of adjustable knobs, arranged in layers, that gets shaped by training data to turn inputs into outputs. The brain metaphor is a marketing artifact from the 1950s that refuses to die.

In 1957, a Cornell psychologist named Frank Rosenblatt built the original perceptron at the Cornell Aeronautical Laboratory. His landmark paper landed in Psychological Review in November 1958, titled "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain."

The thing he built was a single-layer binary classifier. It took weighted inputs, added them up, applied a threshold, and produced a yes or no. It was inspired by what was then known about neurons, but the resemblance was loose even then. Rosenblatt was selling the work to a field that cared about brains.

The name stuck. The metaphor didn't survive contact with the math.

What's Actually Happening

A modern neural network is a stack of layers. Each layer is a bunch of simple units called neurons or nodes. Each unit does the same boring thing. It takes a weighted sum of its inputs, adds a bias, and runs the result through a non-linear function like a sigmoid or a rectified linear unit.

The weights and biases are the parameters. The parameters are the knobs. A large language model today has hundreds of billions of these knobs.

Training is the process of turning the knobs.

You start with a network where every knob is set to a random value. You feed it a training example. It produces a bad output. You measure how bad. Then you nudge every knob a tiny amount in the direction that would have made the output less bad.

You repeat this billions of times.

After enough nudging, the knobs settle into a configuration that produces good outputs on the training data. That's it. That's the magic.

The Algorithm That Makes It Work

The nudging mechanism is called backpropagation. It was popularized in a 1986 Nature paper by David Rumelhart at UC San Diego, Geoffrey Hinton at Carnegie Mellon, and Ronald Williams at Northeastern. The title was "Learning Representations by Back-Propagating Errors." Three pages. It changed everything.

The idea is mechanical. You compute how wrong the output is. You propagate that error signal backward through the layers. Each weight gets adjusted in proportion to how much it contributed to the error. The math is calculus. The chain rule from a college calculus class is the engine that runs modern AI.

Paired with backpropagation is gradient descent. It's the optimization method that tells you which direction to nudge each knob. Picture the loss as a landscape with hills and valleys. Gradient descent rolls the ball downhill.

That's deep learning. A stack of matrix multiplications, non-linear functions in between, gradient descent on top, an error signal flowing backward to update the weights. Repeat for a billion training examples on a GPU cluster.

Where The Brain Metaphor Breaks

Real neurons fire in spikes. They have dendrites, axons, neurotransmitters, complex temporal dynamics. They are wet, three-dimensional, embedded in a metabolism, using over a hundred different neurotransmitters for different jobs.

Artificial neurons are floating-point numbers being multiplied and summed by a GPU.

Backpropagation has no known biological analog. The brain almost certainly does not propagate gradients backward through its synapses the way PyTorch does. Goodfellow, Bengio, and Courville devote a section in chapter one of their 2016 textbook Deep Learning to this exact point. They write that modern deep learning is "not an attempt to model the brain." The comparison persists because it's convenient shorthand, and because that's how the field marketed itself for fifty years.

This matters because the brain metaphor makes AI feel more mysterious than it is. It isn't. It's a function. A giant one with a lot of knobs. The knobs got turned by an algorithm that depends on calculus you could understand in a semester.

The Best Places To Actually See It

Michael Nielsen wrote a free online book called Neural Networks and Deep Learning in 2015. He builds the math step by step by teaching a network to recognize handwritten digits. It's the clearest written treatment I've come across.

Grant Sanderson's 3Blue1Brown channel has a four-part visual series on neural networks. Same handwritten-digit example. Animations that make the matrix math feel intuitive. Over five million people subscribe to that channel. A lot of self-taught engineers started there.

Both make the same move. They strip away the brain language and show the network as a stack of matrix multiplications and non-linear transformations. Once that picture is in place, the rest of deep learning makes sense as engineering, not biology.

The interesting thing isn't that neural networks are like brains. The interesting thing is that a pile of weighted sums and non-linear functions, given enough data and compute, can learn to do things that look like thought. That's worth being curious about on its own.

Sources

• Rosenblatt, Frank. "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain." Psychological Review, Vol 65, No 6, 1958. (opens in new tab)
• Rumelhart, Hinton, and Williams. "Learning Representations by Back-Propagating Errors." Nature, Vol 323, 1986. (opens in new tab)
• Goodfellow, Bengio, and Courville. "Deep Learning" (MIT Press, 2016). (opens in new tab)
• Nielsen, Michael. "Neural Networks and Deep Learning" (Determination Press, 2015). (opens in new tab)
• Sanderson, Grant (3Blue1Brown). "Neural Networks" video series. (opens in new tab)
• Krizhevsky, Sutskever, and Hinton. "ImageNet Classification with Deep Convolutional Neural Networks" (2012, NeurIPS). (opens in new tab)
• Russell and Norvig. "Artificial Intelligence: A Modern Approach" (4th edition, Pearson, 2020). (opens in new tab)

Part of the AI Foundations series. Previous: Machine Learning Is The Part That Learns.