Chrysolites Tech Academy

Applied Deep Learning · Course book

Chapter 3: How a Self-Driving Car Sees

Pairs with Lesson 3: Perception for Self-Driving Cars

A self-driving car must answer three questions continuously: where is the road, what is around me, and where is each of those things going? The part of the system that answers them is called perception, and it turns raw data from cameras and laser scanners into a description of the scene that planning and control can act on. This chapter surveys the main perception tasks and the deep learning methods used for each, as a series of projects on real driving datasets would: segmenting the road, detecting objects in 2D, tracking them over time, working with 3D laser data and the transformations between sensors, learning several tasks with one network, detecting objects in 3D and producing a bird's-eye view. Each section includes a calculation you can do by hand.

Road segmentation with fully convolutional networks

Semantic segmentation labels every pixel of an image with a class, such as road or not road. An ordinary classification network ends with dense layers that throw away spatial layout, so segmentation uses a fully convolutional network (FCN), which contains only convolutional layers. The first half, the encoder, shrinks the image while extracting features; the second half, the decoder, enlarges the feature maps back to the original size with transposed convolutions. Skip connections add detailed maps from early encoder layers to the decoder, so that boundaries come out sharp instead of blurry. The output has one score per class for every pixel.

Segmentation quality is usually measured with intersection over union (IoU) per class: the number of pixels predicted as road that really are road, divided by the number of pixels that are road in either the prediction or the truth. If 9,000 pixels are correctly labelled road, 1,000 are wrongly labelled road and 2,000 road pixels were missed, the IoU is 9,000 / (9,000 + 1,000 + 2,000) = 0.75. Pixel accuracy alone can mislead, because large easy regions such as sky dominate it. Averaging the IoU over all classes, the mean IoU, gives each class an equal say, however small its objects are.

Detecting and tracking objects

Object detection finds each object of interest and draws a box around it, with a class and a confidence score. YOLO, short for "you only look once", does this in a single pass of a network: it divides the image into a grid, and each grid cell predicts a few boxes, each with its position, size, confidence and class probabilities. Because the whole image is processed at once, YOLO is fast enough for video. Many overlapping boxes are predicted for the same object, so non-maximum suppression keeps the most confident box and discards others that overlap it too much, measured again by IoU.

Worked example. Box A runs from (2, 2) to (6, 6) and box B from (4, 3) to (8, 7). Each has an area of 4 × 4 = 16. Their overlap runs from (4, 3) to (6, 6), an area of 2 × 3 = 6. The union is 16 + 16 − 6 = 26, so the IoU is 6 / 26 = 0.23. With a typical suppression threshold of 0.5, both boxes are kept as separate objects; if the overlap had been 12, the IoU would be 12 / 20 = 0.6 and the less confident box would be removed.

Detection treats each frame on its own, but driving needs to know that the car seen now is the same car seen a moment ago. Tracking links detections across frames and gives each object a lasting identity. Deep SORT combines three ideas. A Kalman filter predicts where each tracked object should be in the next frame from its recent motion. The Hungarian algorithm matches new detections to existing tracks so as to minimise the total cost of all matches. And the cost combines a motion distance, the Mahalanobis distance between predicted and detected positions, which accounts for how uncertain the prediction is, with an appearance distance, the cosine distance between feature vectors from a small network that describes what each object looks like. Appearance helps keep identities correct when objects cross or are briefly hidden.

The Kalman filter's central idea is a weighted average between prediction and measurement, with weights set by their uncertainties. In one dimension, with a prediction of variance P and a measurement of variance R, the gain is K = P / (P + R), and the new estimate is the prediction plus K times the difference.

true position 10 7.58.59.5 0246 frame estimate measurements
Figure 3.1 — A one-dimensional Kalman filter. Starting from a poor guess of 8 with variance 1, and measurements of variance 1, the gain falls from 0.5 to 0.14 as confidence grows, so noisy readings move the estimate less and less.

Working the first steps: the starting estimate 8 has variance 1, and the first measurement 10.4 also has variance 1, so K = 0.5 and the estimate becomes 8 + 0.5 × 2.4 = 9.2, with variance 0.5. The second measurement, 9.6, gets a gain of 0.5 / 1.5 = 0.333, giving 9.2 + 0.333 × 0.4 = 9.333. Each step the filter trusts itself a little more and the measurements a little less.

3D data and homogeneous transformations

Many self-driving cars carry a LIDAR sensor, which spins laser beams and measures how long their reflections take to return, producing a point cloud of hundreds of thousands of 3D points per second. Combining it with cameras requires converting coordinates between sensors, each of which has its own frame of reference. Homogeneous coordinates make this tidy: a point (x, y) is written (x, y, 1), so that rotation and translation together become a single matrix multiplication, and several transformations chain by multiplying their matrices.

For example, rotating by 90 degrees and then shifting by (1, 2) uses the matrix with rows (0, −1, 1), (1, 0, 2) and (0, 0, 1). Applied to the point (3, 0, 1), it gives (0 × 3 − 1 × 0 + 1, 1 × 3 + 0 + 2, 1) = (1, 5, 1), so the point lands at (1, 5). The same idea in 3D uses 4 × 4 matrices. Projecting a 3D point onto the camera image uses the camera's focal length and centre: with a focal length of 700 pixels and an image centre at x = 600, a point 2 metres to the right and 20 metres ahead appears at 700 × 2 / 20 + 600 = 670. Driving datasets supply these calibration matrices so that laser points can be drawn on camera images and 3D boxes projected into photos.

Cameras and LIDAR have complementary strengths. Cameras are cheap, see colour and texture, and can read signs and traffic lights, but they measure distance only indirectly and struggle in darkness and glare. LIDAR measures distance directly and precisely in any lighting, but it is more expensive, sees no colour and returns sparser data for objects far away. Some companies rely mainly on cameras and estimate depth with neural networks, while others combine both. Sensor fusion, merging the information from several sensors into one consistent picture, is only possible because the transformations between their frames of reference are known precisely, which is why calibration is checked so carefully.

Multi-task learning, 3D detection and bird's-eye views

A car needs several outputs from the same camera image, such as segmentation and depth. Training separate networks wastes computation, so multi-task learning shares one network for several tasks. The Multi-Task Attention Network (MTAN) uses a shared backbone and, for each task, small attention modules that learn which shared features matter for that task. The total loss adds the task losses, and balancing their weights is part of the design.

TaskInputOutputMethod in the course
Road segmentationCamera imageRoad label per pixelFully convolutional network
2D detectionCamera imageBoxes with classesYOLO
TrackingDetections over timeBoxes with lasting identitiesDeep SORT
Multi-task perceptionCamera imageSegmentation and depthMTAN
3D detectionLIDAR point cloud3D boxes with orientationSFA3D
Bird's-eye viewSeveral camera imagesTop-down map of the sceneUNetXST

3D object detection finds objects as boxes in space, with position, size and heading. SFA3D, a fast single-stage method, first converts the point cloud into a top-down image whose pixels record height, intensity and density of points, then detects objects in that image with a convolutional network, which is much quicker than processing raw points. Finally, planning is easiest on a bird's-eye view, a map seen from above. UNetXST learns to transform several camera images into one top-down semantic map, using known camera geometry to guide a U-shaped encoder-decoder network. Together these tasks give the planner a picture of the road, the objects on it and where each is heading.

Check your understanding

  1. A road prediction has 7,200 correct road pixels, 800 false road pixels and 1,000 missed road pixels. Compute the IoU.
  2. Compute the IoU of boxes (0, 0)–(4, 4) and (2, 2)–(6, 6). Would non-maximum suppression with a threshold of 0.5 remove one?
  3. Continue the Kalman example from estimate 9.333 with variance 0.333 and a new measurement of 10.3. Give the gain and the new estimate.
  4. Apply the rotate-and-shift matrix from this chapter to the point (0, 2).
  5. Why does Deep SORT use appearance features as well as predicted motion?
  6. Explain why multi-task learning can be more efficient than separate networks, and one difficulty it introduces.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.