Chrysolites Tech Academy

Deep Learning with TensorFlow and Keras · Course book

Chapter 2: Image and Text Classifiers in Keras

Pairs with Lesson 2: TensorFlow 2 Crash Course

The fastest way to learn a deep learning library is to build two small but complete projects with it, one for images and one for text, and to understand every line along the way. This chapter does exactly that with TensorFlow 2 and Keras. It starts with a compact explanation of what a neural network is, then walks through loading and inspecting a dataset, defining a model, training it and using it to make predictions. The second project classifies film reviews as positive or negative, which introduces the techniques every text model needs: turning words into integers, padding sequences to a common length, learning word embeddings and averaging them into a single vector. The chapter ends with saving models and with what is involved in running TensorFlow on a graphics card.

What a neural network is, in one page

A neural network is a function with many adjustable numbers. Data enters at the input layer, one neuron per value. Each neuron in the next layer computes a weighted sum of the neurons feeding it, adds a bias and applies an activation function. The final output layer gives the answer: for a choice between ten classes, ten neurons whose values are turned into probabilities by softmax; for a yes-or-no question, a single neuron squashed between 0 and 1 by the sigmoid.

At first the weights are random, so the outputs are nonsense. Training feeds in examples whose answers are known, measures the error with a loss function, and uses an optimiser to nudge every weight in the direction that reduces the loss. Each nudge is tiny; thousands of them, over many examples, turn random weights into a useful model. The key point for using a library is that you choose the architecture, the loss and the optimiser, and the library handles the calculus.

Load andinspect Preprocessand split Definelayers Compileand fit Evaluateand predict not good enough? change the model and try again
Figure 2.1 — The workflow shared by both projects in this chapter. The loop from evaluation back to the model is where most of the real work happens.

Loading, inspecting and modelling image data

Keras ships several practice datasets that download with a single call and arrive already split into training and test sets. The clothing image set used here returns 60,000 training images and 10,000 test images, each a 28 × 28 grid of brightness values from 0 to 255, with labels from 0 to 9. The labels are just integers, so a short list of class names, such as "T-shirt", "Trouser" and "Ankle boot", is kept alongside to make the results readable.

Inspecting data before training is never wasted. Printing the shape of the arrays confirms there are 60,000 images of 28 × 28. Displaying a few images with their labels catches mistakes such as mismatched labels. Printing one image's raw values shows the 0 to 255 range, which leads to the first preprocessing step: dividing every pixel by 255 so that inputs lie between 0 and 1. Large, uneven input values make training slower and less stable.

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dense(10, activation="softmax"),
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.fit(train_images, train_labels, epochs=5)
test_loss, test_acc = model.evaluate(test_images, test_labels)

The model flattens each image into 784 values, passes them through a hidden layer of 128 ReLU neurons and ends with 10 softmax outputs. The loss name begins with "sparse" because the labels are plain integers rather than one-hot vectors. Each epoch is one pass over all 60,000 images; with the default batch size of 32, that is 60,000 / 32 = 1,875 weight updates per epoch. Training accuracy usually ends a little above test accuracy, and a large gap between the two is the sign of overfitting.

To make a prediction, call model.predict on a batch of images. It returns, for each image, ten probabilities that sum to 1. The index of the largest, found with NumPy's argmax, is the predicted class, which you look up in the list of names. Showing the image, the predicted name and the true name side by side for a handful of test images is the quickest honest check of what the model has really learned.

Preparing text for a network

Text needs more preparation than images, because networks only accept numbers and because sentences differ in length. The film review dataset in Keras contains 50,000 reviews, split evenly into 25,000 for training and 25,000 for testing, each labelled positive or negative. The reviews arrive already encoded: each word has been replaced by an integer, based on its rank by frequency, so lower codes stand for more common words. Keeping only the most frequent 10,000 words is a common choice, and rarer words map to a shared "unknown" code.

A small table of special codes is usually reserved at the start of the vocabulary.

CodeMeaningWhy it is needed
0PaddingFills short reviews up to the fixed length
1StartMarks the beginning of each review
2UnknownStands in for words outside the kept vocabulary
3UnusedReserved, so real words begin at code 4

To read an encoded review, you invert the word index, a dictionary from words to codes, into one from codes to words, remembering to shift every code by the three reserved values. Decoding a few reviews this way is a valuable check that the data really is what you think it is.

A network needs every input to be the same length, so reviews are padded or truncated. With a fixed length of 8, the encoded review [1, 14, 22, 16, 43] becomes [1, 14, 22, 16, 43, 0, 0, 0] with padding added at the end, while a review of 12 codes is cut to its first 8. A real model would use a much larger length, often in the low hundreds, chosen so that most reviews fit without losing much text.

Embeddings and a text classifier

An integer code says nothing about meaning: code 14 is not "close to" code 15 in any useful sense. An embedding layer replaces each code with a learned vector, for example of 16 numbers. During training, words used in similar ways, such as "great" and "excellent", drift towards similar vectors because they push the prediction in the same direction.

The classifier then needs one vector per review, however many words it has. A global average pooling layer averages the word vectors. With three words whose two-number embeddings are (0.2, 0.8), (0.6, −0.4) and (1.0, 0.2), the average is ((0.2 + 0.6 + 1.0) / 3, (0.8 − 0.4 + 0.2) / 3) = (0.6, 0.2). That single vector passes through a small dense layer and a final sigmoid neuron that outputs the probability that the review is positive.

Worked example. Count the parameters for a vocabulary of 10,000 words, 16-number embeddings, average pooling, a dense layer of 16 ReLU neurons and one sigmoid output. The embedding holds 10,000 × 16 = 160,000 values. Pooling has no parameters. The dense layer has 16 × 16 + 16 = 272, and the output 16 × 1 + 1 = 17. The total is 160,000 + 272 + 17 = 160,289, and 99.8 percent of them live in the embedding table, which is typical for text models.

The loss for a single yes-or-no output is binary cross-entropy. Part of the training set, say 10,000 reviews, is held back as a validation set and passed to fit, so that each epoch reports both training and validation accuracy. When validation accuracy stops improving while training accuracy keeps rising, more epochs will only memorise the training reviews. New text, such as a review you write yourself, must go through exactly the same steps before prediction: the same word index, the same reserved codes and the same padding length.

Saving models and using a GPU

Training a model can take a long time, so you should not repeat it every time a program starts. model.save writes the architecture, the learned weights and the optimiser state to a single file, and keras.models.load_model restores it, ready to predict. A common pattern is a training script that saves the model and a separate, quick prediction script that loads it.

Neural networks spend most of their time multiplying large matrices, work that graphics processing units do in parallel far faster than ordinary processors. Using one with TensorFlow on Linux needs several pieces that must agree with one another: a supported graphics driver, the vendor's computing toolkit, a matching deep learning library for it, and a TensorFlow build compatible with those versions. Version mismatches are the most common cause of trouble, so the safest route is to follow the compatibility table in the official installation guide for the TensorFlow version you are using. After installation, asking TensorFlow to list its physical GPU devices confirms whether it can see the card. If it cannot, everything still works on the processor, only more slowly.

Practice tasks

  1. With 60,000 training images and a batch size of 64, how many weight updates happen in one epoch, rounding up for the last partial batch?
  2. Pad the encoded review [1, 7, 9] to length 6 with padding at the end, then truncate [1, 5, 8, 2, 9, 4, 6, 3] to length 6 keeping the beginning.
  3. Average the embeddings (1.0, 0.0), (0.0, 1.0), (0.5, 0.5) and (0.5, −0.5).
  4. Count the parameters of a text model with a vocabulary of 20,000, 32-number embeddings, average pooling, a dense layer of 24 neurons and one output.
  5. Explain why new text must be encoded with exactly the same word index and padding length used in training.
  6. List what model.save stores and describe a situation where you would load a saved model instead of training again.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.