Chrysolites Tech Academy

Video by freeCodeCamp.org · streamed from YouTube. All rights remain with the creator.

Lesson 1: The Ideas That Built Deep Learning

About this lesson

This lesson is an unusually deep course: more than twenty-seven hours that teach modern neural networks by rebuilding, one after another, the landmark results from Geoffrey Hinton's research career. The material runs in historical order, so you see each idea arrive as the answer to a problem that the previous methods could not solve. It starts with Boltzmann machines and then backpropagation, the procedure that still trains almost every network today. It moves on to deep belief networks, which first made many-layered models trainable, and to t-SNE, a method for drawing high-dimensional data on a flat page. Next comes AlexNet, the image model whose success in 2012 changed the direction of the field, followed by dropout for preventing overfitting and knowledge distillation for compressing a large model into a small one. The later sections cover capsule networks, layer normalization and the forward-forward algorithm, a recent proposal for learning without a backward pass. Every topic follows the same pattern: the motivation for the work, the problem it addressed and its central idea, and then an implementation in PyTorch that connects the theory to running code. You should be comfortable with Python and have some earlier exposure to neural networks before starting. Treat it as a reference to return to, taking one paper at a time. Afterwards, read Chapter 1 of the course book, which summarises the whole sequence in a single table and works dropout, layer normalization and softmax temperature by hand.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.