Video by freeCodeCamp.org · taught by Aakash N S · streamed from YouTube. All rights remain with the creator.
Lesson 6: ResNets, Regularisation and Data Augmentation
About this lesson
This fifth session of the Deep Learning with PyTorch live course, taught by Aakash N S, shows how to turn an ordinary convolutional network into a highly accurate image classifier by combining a set of proven techniques. Working again with the CIFAR-10 dataset of small colour photographs, it improves three things in turn: the data, the model and the training process. For the data, the session explains how to normalise each colour channel using statistics from the training images, and how data augmentation, such as random crops and horizontal flips, creates fresh variations of the training images so that the network overfits less. For the model, it introduces residual connections, which add a block's input to its output and make deeper networks much easier to train, together with batch normalisation, which keeps the values flowing through the network in a stable range. For training, it covers learning rate scheduling with the one-cycle policy, which raises and then lowers the learning rate during training, along with weight decay to discourage large weights and gradient clipping to stop occasional very large gradients from disrupting the weights. The session then puts everything together and trains a residual network from scratch, reaching a high accuracy within minutes on a GPU. Plotting the accuracy, the losses and the learning rate shows how the techniques work together. The finished model is tested on individual images and saved. Afterwards, read Chapter 6 of the course book, which works batch normalisation by hand.