Chrysolites Tech Academy

Deep Learning with PyTorch · Course book

Chapter 7: Generative Adversarial Networks

Pairs with Lesson 7: GANs for Image Generation

Every model so far has been discriminative: given an image, it predicts a label. Generative models do the reverse, producing new examples that look as if they came from the training data. Generative adversarial networks, or GANs, were one of the first approaches to produce convincing images, and the idea behind them is a contest between two networks. One, the generator, tries to create realistic fakes; the other, the discriminator, tries to tell fakes from real images. Each improves by learning from the other. This chapter explains that contest, the losses that drive it, the architecture of both networks and the training loop that alternates between them, with every loss and shape worked out by hand, and closes with the common difficulties of training GANs and how to judge their results.

Generative modelling and GANs

A generative model learns the patterns in a collection of examples well enough to produce new ones. For images of handwritten digits, that means learning which arrangements of strokes look like real handwriting, so that it can draw a digit nobody actually wrote. Applications include creating artwork and textures, filling in missing parts of photos, increasing image resolution, generating extra training data and turning sketches into pictures. The same power brings responsibility: realistic generated images can mislead, so it is good practice to label generated content clearly and to use these tools only in honest, respectful ways.

A GAN has two parts. The generator takes a vector of random numbers, called the latent vector, for example 64 values drawn from a normal distribution, and transforms it into an image. Different random vectors give different images. The discriminator is an ordinary binary classifier: it takes an image and outputs the probability that the image is real rather than generated. The two are trained together, the discriminator to catch fakes and the generator to fool the discriminator.

randomvector Generator fake images real images Discriminator P(real) feedback: how convincing were the fakes?
Figure 7.1 — The two networks of a GAN. The discriminator sees real and generated images; its judgements are the training signal for both networks.

The adversarial losses

Both networks are trained with binary cross-entropy. The discriminator wants to output 1 for real images and 0 for fakes, so its loss is −ln D(real) for each real image plus −ln(1 − D(fake)) for each generated one. The generator wants the discriminator to output 1 for its fakes, so its loss is −ln D(G(z)), where G(z) is the image made from latent vector z. The generator never sees a real image directly; everything it learns comes through the discriminator's gradients.

Worked example. The discriminator gives a real image a probability of 0.8 of being real, and a generated image 0.3. Its loss is −ln 0.8 − ln(1 − 0.3) = 0.223 + 0.357 = 0.580. The generator's loss for the same fake is −ln 0.3 = 1.204: the discriminator was not fooled, so the generator has a large incentive to improve. If the generator became perfect, the best the discriminator could do is guess 0.5 for everything; then its loss would be 2 × −ln 0.5 = 1.386 and the generator's would be −ln 0.5 = 0.693.

This is a two-player game rather than ordinary minimisation. A falling discriminator loss means it is winning; a falling generator loss means the generator is winning. Healthy training usually shows both losses fluctuating within a range rather than either one steadily reaching zero. If the discriminator becomes far too strong too early, the generator's gradients become very small and it stops learning, which is one reason the two are trained in balance.

The generator's loss is written as −ln D(G(z)) for a practical reason. The original formulation of the game had the generator minimise ln(1 − D(G(z))), the opposite of what the discriminator maximises. Early in training, though, the discriminator easily rejects the poor fakes, D(G(z)) is close to 0, and that expression is almost flat there, so the generator receives hardly any gradient. Changing its objective to making D(G(z)) large, by minimising −ln D(G(z)), gives the same goal but a steep slope exactly when the generator is doing badly. At D(G(z)) = 0.05, for example, the loss is −ln 0.05 = 3.0 and still falling steeply, which pushes the generator hard to improve.

Building the two networks

For 28 × 28 digit images, both networks can be small stacks of linear layers. Real images are first scaled from the range 0 to 1 into −1 to 1, by multiplying by 2 and subtracting 1, so a pixel of 0.25 becomes −0.5. The generator's last activation is then tanh, whose outputs also lie between −1 and 1, so real and fake images share the same range. Hidden layers commonly use leaky ReLU, which lets a small gradient through for negative inputs and helps the generator receive useful feedback.

NetworkLayersOutputParameters
Generator64 → 256 → 256 → 784, leaky ReLU, then tanhA 28 × 28 image in −1 to 116,640 + 65,792 + 201,488 = 283,920
Discriminator784 → 256 → 256 → 1, leaky ReLU, then sigmoidProbability the image is real200,960 + 65,792 + 257 = 267,009

For larger colour images, convolutional versions work far better. The discriminator becomes an ordinary convolutional classifier that shrinks the image with strided convolutions. The generator runs the process in reverse with transposed convolutions, which enlarge a feature map. For input size n, kernel k, stride s and padding p, the output size is (n − 1) × s − 2p + k. Starting from the latent vector treated as a 1 × 1 map, a layer with k = 4, s = 1 and p = 0 gives 4 × 4; then three layers with k = 4, s = 2 and p = 1 give 8, 16 and finally 32 pixels wide, a full image grown from a single point. Batch normalisation between the layers helps keep both networks stable.

The latent vector is more than a source of randomness. After training, nearby vectors produce similar images, and moving steadily from one vector to another produces a smooth sequence of images in between. The generator has, in effect, organised a compact space of 64 numbers into a map of plausible digits. Some directions in that space can even line up with recognisable properties, such as stroke thickness or slant, although nothing in the training asked for that explicitly.

Training the networks together

Each training step works on one batch of real images and has two phases. In the first, the discriminator is updated. It is shown the real batch with target labels of 1, and a batch of fakes, made by passing fresh random vectors through the generator, with target labels of 0. The two losses are added, backpropagated, and the discriminator's optimiser takes a step. In the second phase, the generator is updated. A new batch of fakes is created and passed to the discriminator, but this time the target labels are 1, because the generator wants them judged real. The resulting loss is backpropagated through the discriminator into the generator, and only the generator's optimiser takes a step.

for real_images, _ in loader:
    # 1. train the discriminator
    d_loss = bce(D(real_images), ones) + bce(D(G(noise()).detach()), zeros)
    d_opt.zero_grad(); d_loss.backward(); d_opt.step()

    # 2. train the generator
    g_loss = bce(D(G(noise())), ones)
    g_opt.zero_grad(); g_loss.backward(); g_opt.step()

The call to detach() in the first phase stops the discriminator's loss from changing the generator. Each network has its own optimiser, often Adam with a learning rate around 0.0002. To watch progress, a fixed set of latent vectors is chosen at the start and passed through the generator after every epoch, and the resulting grid of images is saved. Because the inputs stay the same, the saved grids show the same "seeds" gradually turning from noise into recognisable digits, and stitching them together gives a short animation of the generator learning.

Difficulties and evaluation

GANs are notoriously delicate to train. In mode collapse the generator finds a few outputs that fool the discriminator and produces only those, for example drawing only 1s, so the images look plausible but lack variety. Training can also oscillate, with each network repeatedly overpowering the other, or fail to start if the discriminator wins so decisively that the generator's gradients vanish. Common remedies include balancing the learning rates, using leaky ReLU and batch normalisation, adding a little noise or softening the real labels from 1 to about 0.9, and monitoring sample grids as well as the losses.

Judging a generative model is harder than judging a classifier, because there is no single correct output. The most direct check is to look: are the samples sharp, varied and realistic? Interpolating between two latent vectors and generating the images in between is another revealing test; a good generator changes smoothly from one digit to another. More formal scores compare statistics of generated and real images computed with a pre-trained network, but visual inspection remains essential.

Check your understanding

  1. The discriminator gives a real image 0.9 and a fake 0.2. Compute the discriminator's loss and the generator's loss for that fake.
  2. Why is the generator's last activation tanh, and what must be done to the real images to match it?
  3. Compute the output size of a transposed convolution with n = 7, k = 4, s = 2 and p = 1.
  4. Count the parameters of a generator with layers 100 → 128 → 784.
  5. Explain the purpose of detach() when training the discriminator on fake images.
  6. Describe mode collapse and suggest one way to detect it from the saved sample grids.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.