Chrysolites Tech Academy

How Neural Networks Work · Course book

Chapter 2: The Language of Deep Learning

Pairs with Lesson 2: Deep Learning Key Concepts

Deep learning comes with a large vocabulary, and much of the difficulty for newcomers is simply the terms: activation, loss, optimiser, epoch, batch, hyperparameter, dropout. Each one names a simple idea, and once the ideas are connected, papers and tutorials become far easier to follow. This chapter is a guided map of that vocabulary. It explains what deep learning is and how a network learns, then goes through the core terms one family at a time with small calculations to make each concrete. It continues with the three kinds of learning, the techniques used to stop a network memorising its training data, and the main network architectures. It ends with a five-step recipe that applies to every deep learning project, whatever its size.

What deep learning is and how networks learn

Artificial intelligence is the broad goal of making machines behave intelligently. Machine learning is one approach to it, in which a system learns from data rather than following hand-written rules. Deep learning is a branch of machine learning that uses neural networks with many layers. Its key advantage is that it learns its own features: instead of a person deciding that the edges and colours of an image matter, the early layers discover useful patterns and later layers combine them. That ability is why deep learning thrives when there is a great deal of data and computing power, and why it often struggles when there is little.

A neural network is organised in layers. The input layer receives the data, one value per feature or pixel. One or more hidden layers transform it. The output layer produces the prediction. Each connection between neurons has a weight, and each neuron has a bias. Learning has two phases that repeat. In forward propagation, data flows through the network to produce a prediction. In backpropagation, the error in that prediction flows backwards, telling every weight how it should change. An optimiser then makes the change. Repeat this enough times over enough examples and the predictions improve.

The core terminology

Activation functions decide what each neuron passes on. Without them a network of any depth would behave like a single linear equation. The common choices are summarised in the table.

ActivationOutput rangeTypical useNote
Step0 or 1Historical perceptronsNo useful slope, so it cannot be trained by gradients
Sigmoid0 to 1Output for yes-or-no problemsSlope fades for large inputs
Tanh−1 to 1Hidden layers in older and recurrent networksCentred on zero
ReLU0 upwardsDefault for hidden layersFast; neurons can get stuck at zero
Leaky ReLUAny valueHidden layersSmall slope for negatives keeps neurons alive
Softmax0 to 1, summing to 1Output for choosing one of several classesTurns scores into probabilities

Softmax deserves a calculation. Suppose the output layer gives raw scores of 2.0, 1.0 and 0.1 for three classes. Softmax raises e to each score, giving 7.389, 2.718 and 1.105, adds them to get 11.213, and divides each by the total. The probabilities are 0.659, 0.242 and 0.099, which add up to 1, and the biggest score keeps the biggest share.

Loss functions measure how wrong the network is. For predicting numbers, mean squared error averages the squared gaps. For choosing classes, cross-entropy takes the negative logarithm of the probability given to the correct class. Using the softmax output above, if the correct class is the first, the loss is −ln 0.659 = 0.417; if it is the second, the loss is −ln 0.242 = 1.419. A network that gave the right answer only a small probability pays a much larger penalty.

Optimisers decide how to change the weights once the gradients are known. Plain gradient descent takes a step against the gradient, scaled by the learning rate. Stochastic gradient descent estimates the gradient from a small random batch rather than the whole dataset, which is far cheaper. Momentum keeps a running average of past steps so the optimiser rolls through small bumps. Adagrad, RMSprop and Adam go further and give each weight its own effective learning rate based on the history of its gradients; Adam, which combines momentum with this per-weight scaling, is the usual default.

The learning rate is the single most important setting. The graph below shows the effect of three learning rates when minimising a very simple loss, the square of a single weight starting at 1. Each step multiplies the weight by one minus twice the learning rate.

0123 0246 steploss 1.05: diverges 0.05: slow 0.4: fast
Figure 2.1 — Loss over six steps for three learning rates on the loss w2. Too small is slow, a good value converges quickly, and too large makes the loss grow.

That leads to the difference between parameters and hyperparameters. Parameters, the weights and biases, are learned from the data. Hyperparameters are chosen by you before training: the learning rate, the number and size of layers, the batch size, the number of epochs and the activation functions. Finding good hyperparameters is done by experiment, guided by a validation set.

Finally, three words describe how training is scheduled. A batch is the group of examples processed before each weight update. An iteration is one such update. An epoch is one full pass through the training data. With 50,000 training examples and a batch size of 100, one epoch takes 50,000 / 100 = 500 iterations, and ten epochs take 5,000.

Kinds of learning

Supervised learning uses labelled examples: images tagged with what they show, or emails marked as spam or not. The network learns to map inputs to labels, and it covers both classification and regression. Unsupervised learning has no labels and looks for structure: grouping similar customers, compressing data, or spotting unusual records that do not fit the usual patterns. Reinforcement learning involves an agent acting in an environment, receiving rewards or penalties and learning a strategy that earns the most reward over time; it has been used to master board games and to control simulated robots. Many real systems mix the three, for example learning general features without labels and then fine-tuning on a smaller labelled set.

Regularisation: fighting overfitting

Deep networks have so many parameters that they can memorise their training data, a failure called overfitting. The warning sign is a training loss that keeps falling while the validation loss starts rising. Regularisation is the family of techniques that push the network towards general patterns instead.

  • Dropout switches off a random fraction of neurons, often half, at each training step, so the network cannot rely on any single neuron and must spread what it learns. At prediction time all neurons are used.
  • Data augmentation creates extra training examples by transforming existing ones, for example flipping, rotating, cropping or recolouring images, so the network sees more variety.
  • Early stopping watches the validation loss and stops training when it stops improving, keeping the best weights seen so far.
  • Weight penalties, such as L2 regularisation, add the size of the weights to the loss, favouring smaller, smoother solutions.

The most reliable cure of all is more real data, which is why collecting and cleaning data is usually the best use of a project's time.

Network architectures and the five steps

A fully connected feedforward network connects every neuron in one layer to every neuron in the next, with data flowing only forwards. It suits tables of features. A recurrent network loops information from one step to the next and suits sequences such as text and time series; its LSTM and GRU variants add gates that help it remember over long spans. A convolutional network slides small learned filters across a grid such as an image, finding patterns wherever they occur, and dominates computer vision.

Five steps to every deep learning model. 1. Gather data: enough, relevant and correctly labelled examples. 2. Preprocess: clean it, handle missing values, scale features and split it into training, validation and test sets. 3. Train: choose an architecture, a loss and an optimiser, and fit the model to the training set. 4. Evaluate: measure performance on the validation and test sets with a metric that reflects the real goal. 5. Optimise: tune hyperparameters, add regularisation or more data, and repeat until the results are good enough.

The recipe is deliberately simple, and that is its value. When a project is going badly, it is almost always possible to point at one step: too little data, leaking test data into preprocessing, a loss that does not match the task, a misleading metric, or tuning against the test set until it stops being a fair judge.

Check your understanding

  1. Compute the softmax probabilities for scores of 1.0, 1.0 and 0.0, and the cross-entropy loss if the correct class is the third.
  2. A dataset has 12,000 examples and the batch size is 64. How many iterations make up one epoch, rounding up for the final partial batch?
  3. In the learning-rate example, what happens with a learning rate of exactly 0.5, and why?
  4. Sort these into parameters and hyperparameters: a bias, the learning rate, a weight, the number of hidden layers, the dropout rate.
  5. Name the type of learning used in each case: grouping news articles by topic with no labels, predicting house prices from past sales, teaching an agent to balance a pole.
  6. Your validation loss started rising after epoch 8 while training loss kept falling. Name two regularisation techniques you could try and explain how each helps.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.