How Modern AI Models Work · Course book
Chapter 1: Four Decades of Removing Obstacles
Deep learning did not arrive in one leap. It was assembled over four decades from a series of separate ideas, each of which removed one obstacle that stood in the way. Many of those ideas came from Geoffrey Hinton and his students, which is why his papers make such a good map of the field. This chapter walks through the main ideas in order, explains the problem each one solved, and works through the small calculations at the heart of three of them.
Learning by adjusting connections
A neural network is a set of simple units joined by connections, each with a strength called a weight. The founding question of the field is how a network can set its own weights from examples instead of having them set by hand.
One early answer, from 1985, was the Boltzmann machine. Borrowing ideas from physics, it treats a network as a system with an energy, in which likely patterns have low energy and unlikely ones have high energy. Learning lowers the energy of patterns seen in the data. The idea was elegant, and it showed that a network could discover structure without being told the right answers, but training was far too slow for large problems.
The answer that lasted came a year later: backpropagation. A network makes a prediction, the prediction is compared with the correct answer to give an error, and the error is passed backwards through the layers. Using the chain rule from calculus, each weight learns how much it contributed to the error and is nudged in the direction that reduces it. Repeat this over many examples and the network improves. Backpropagation made it possible to train networks with hidden layers, layers between input and output whose units learn their own useful features.
Nearly every modern AI system is still trained this way. Yet for some twenty years it worked well only on small networks. The remaining story is largely about what had to be added before it worked on large ones.
Making deep networks trainable
"Deep" simply means many layers. In principle more layers let a network build more elaborate features, yet for years deep networks trained worse than shallow ones. As the error signal was passed back through many layers it shrank towards nothing, so the early layers barely learned. This is the vanishing gradient problem.
In 2006 came a way round it: the deep belief network. Instead of training all the layers at once, each was trained separately, one after another, to model the output of the layer beneath it, with no labelled answers required. Only then was the whole stack fine-tuned with backpropagation. This layer-by-layer pre-training gave the weights a sensible starting point, and it revived interest in deep networks at a time when most researchers had moved on.
The decisive moment was 2012. A network now known as AlexNet entered the leading image-recognition contest, in which programs must label photographs from a thousand categories. Its error rate on the standard measure was 15.3 percent; the next best entry scored 26.2 percent, a gap of nearly 11 points in a contest usually won by a point or two. Three ingredients made it possible.
- Graphics processors. Training ran on chips built for video games, which perform many simple calculations at once and made a network of about 60 million weights practical.
- A simpler activation function. Each unit used the rule "if the input is negative output zero, otherwise pass it through unchanged", known as ReLU. Unlike earlier smooth functions it does not squash large values, so the error signal survives the journey back through many layers.
- Dropout, described next, which stopped so large a network from merely memorising its training pictures.
With these in place, pre-training was no longer needed, and within a few years deep networks led in speech, translation and vision.
Keeping large networks honest
A network with millions of weights can memorise its training examples perfectly and still fail on new ones, a fault called overfitting. Two techniques from this period remain standard defences and stabilisers.
Dropout
During training, dropout switches off a random selection of units for each example, commonly half of them. No unit can then rely on particular partners being present, so each must learn something useful in its own right. The survivors are scaled up to compensate. Suppose four units output 2, 4, 6 and 8, and the dropout rate is one half.
- A random mask keeps the first and third units and drops the others, giving 2, 0, 6, 0.
- The survivors are divided by the fraction kept, 0.5, which doubles them: 4, 0, 12, 0.
The original values sum to 20. This draw sums to 4 + 12 = 16, and a draw that kept the second and fourth units instead would sum to 8 + 16 = 24. Averaged over all possible masks the total is exactly 20, so the next layer receives signals of the same typical size as it will when nothing is dropped. When the trained network is used, nothing is dropped.
Layer normalization
As values pass through many layers they can drift to very large or very small sizes, which makes training unstable. Layer normalization rescales the outputs of a layer, for each example, so that they have a mean of 0 and a spread of 1. Take the outputs 2, 4, 6 and 8 again.
- The mean is (2 + 4 + 6 + 8) ÷ 4 = 5.
- The differences from the mean are -3, -1, 1 and 3. Their squares are 9, 1, 1 and 9, which average to 5. The standard deviation is the square root of 5, about 2.236.
- Dividing each difference by 2.236 gives -1.34, -0.45, 0.45 and 1.34.
Whatever sizes arrive, the layer hands on well-behaved numbers. Layer normalization is a fixture of the transformer models behind today's language systems.
Smaller models and a view inside
The best networks are often too large and slow to run on a phone. Knowledge distillation, from 2015, trains a small "student" network to imitate a large "teacher". Its insight is that the teacher's full output is more informative than the bare right answer. If the teacher says a photo is 87 percent likely to be a dog, 12 percent a wolf and 2 percent a cat, it is also saying that dogs resemble wolves more than cats, and that knowledge is worth passing on.
To bring out those small probabilities the teacher's outputs are softened with a temperature. A network's raw scores are turned into probabilities by the softmax function: raise e to the power of each score, then divide by the total. With a temperature T, each score is first divided by T. Take raw scores of 4, 2 and 0.
- At T = 1 the powers are e4 = 54.60, e2 = 7.39 and e0 = 1, totalling 62.99. The probabilities are 0.87, 0.12 and 0.02.
- At T = 4 the scores become 1, 0.5 and 0. The powers are 2.72, 1.65 and 1, totalling 5.37. The probabilities are 0.51, 0.31 and 0.19.
The order never changes, but the softened version makes the teacher's opinion of the runners-up plain, and the student learns from all of it.
A different kind of tool from the same years lets people look inside. A network represents each input as a point in a space of hundreds of dimensions, which cannot be drawn. t-SNE, from 2008, lays those points out on a flat page so that points close together in the original space stay close together. The resulting pictures, in which examples of the same kind gather into islands, became the standard way of checking what a network has learned.
Questioning the foundations
Hinton's later work is notable for doubting methods he helped to establish. Capsule networks, from 2017, respond to a weakness in image networks: they detect that an image contains the right parts without checking that the parts are arranged correctly. A capsule is a group of units that records not only whether a feature is present but also its pose, such as position and angle, so that parts must agree with one another before a whole is recognised.
The forward-forward algorithm, from 2022, goes further and proposes an alternative to backpropagation itself. In place of one forward pass and one backward pass it uses two forward passes, one on real data and one on made-up data, with each layer adjusting itself locally to respond strongly to the first and weakly to the second. It is less accurate than backpropagation at present, but it may suit low-power hardware and is closer to how a brain could plausibly learn. Table 1.1 gathers the story together.
| Year | Idea | Problem it addressed |
|---|---|---|
| 1985 | Boltzmann machines | Learning structure from data without labelled answers |
| 1986 | Backpropagation | Training networks that have hidden layers |
| 2006 | Deep belief networks | Getting many-layered networks to train at all |
| 2008 | t-SNE | Seeing high-dimensional representations on a flat page |
| 2012 | AlexNet and dropout | Scaling up on graphics processors without overfitting |
| 2015 | Knowledge distillation | Fitting a large model's ability into a small one |
| 2016 | Layer normalization | Keeping values stable through deep networks |
| 2017 | Capsule networks | Recognising how parts are arranged, not merely present |
| 2022 | Forward-forward algorithm | Learning without a backward pass |
Hinton shared the 2018 Turing Award, computing's highest honour, with Yoshua Bengio and Yann LeCun, and the 2024 Nobel Prize in Physics with John Hopfield. The lesson of the sequence is that progress came from naming an obstacle precisely and removing it, one at a time.
Check your understanding
- What does backpropagation pass backwards through a network, and what is it used for?
- Why did deep networks train poorly before 2006, and how did layer-by-layer pre-training help?
- Units output 3, 5, 7 and 9, and dropout at a rate of one half keeps only the second and fourth. What are the values after scaling? (Answer: 0, 10, 0, 18.)
- Apply layer normalization to the outputs 1, 3, 5 and 7. (Answer: the mean is 4, the squared differences 9, 1, 1 and 9 average to 5, the standard deviation is 2.236, and the results are -1.34, -0.45, 0.45 and 1.34.)
- What happens to softmax probabilities as the temperature rises, and why does that help a student network?
- What does a capsule record that an ordinary unit does not?
This course book is available to read online only at tech-academy.chrysolites.org.