Applied Deep Learning · Course book
Chapter 3: How a Self-Driving Car Sees
A self-driving car must answer three questions continuously: where is the road, what is around me, and where is each of those things going? The part of the system that answers them is called perception, and it turns raw data from cameras and laser scanners into a description of the scene that planning and control can act on. This chapter surveys the main perception tasks and the deep learning methods used for each, as a series of projects on real driving datasets would: segmenting the road, detecting objects in 2D, tracking them over time, working with 3D laser data and the transformations between sensors, learning several tasks with one network, detecting objects in 3D and producing a bird's-eye view. Each section includes a calculation you can do by hand.
Road segmentation with fully convolutional networks
Semantic segmentation labels every pixel of an image with a class, such as road or not road. An ordinary classification network ends with dense layers that throw away spatial layout, so segmentation uses a fully convolutional network (FCN), which contains only convolutional layers. The first half, the encoder, shrinks the image while extracting features; the second half, the decoder, enlarges the feature maps back to the original size with transposed convolutions. Skip connections add detailed maps from early encoder layers to the decoder, so that boundaries come out sharp instead of blurry. The output has one score per class for every pixel.
Segmentation quality is usually measured with intersection over union (IoU) per class: the number of pixels predicted as road that really are road, divided by the number of pixels that are road in either the prediction or the truth. If 9,000 pixels are correctly labelled road, 1,000 are wrongly labelled road and 2,000 road pixels were missed, the IoU is 9,000 / (9,000 + 1,000 + 2,000) = 0.75. Pixel accuracy alone can mislead, because large easy regions such as sky dominate it. Averaging the IoU over all classes, the mean IoU, gives each class an equal say, however small its objects are.
Detecting and tracking objects
Object detection finds each object of interest and draws a box around it, with a class and a confidence score. YOLO, short for "you only look once", does this in a single pass of a network: it divides the image into a grid, and each grid cell predicts a few boxes, each with its position, size, confidence and class probabilities. Because the whole image is processed at once, YOLO is fast enough for video. Many overlapping boxes are predicted for the same object, so non-maximum suppression keeps the most confident box and discards others that overlap it too much, measured again by IoU.
Detection treats each frame on its own, but driving needs to know that the car seen now is the same car seen a moment ago. Tracking links detections across frames and gives each object a lasting identity. Deep SORT combines three ideas. A Kalman filter predicts where each tracked object should be in the next frame from its recent motion. The Hungarian algorithm matches new detections to existing tracks so as to minimise the total cost of all matches. And the cost combines a motion distance, the Mahalanobis distance between predicted and detected positions, which accounts for how uncertain the prediction is, with an appearance distance, the cosine distance between feature vectors from a small network that describes what each object looks like. Appearance helps keep identities correct when objects cross or are briefly hidden.
The Kalman filter's central idea is a weighted average between prediction and measurement, with weights set by their uncertainties. In one dimension, with a prediction of variance P and a measurement of variance R, the gain is K = P / (P + R), and the new estimate is the prediction plus K times the difference.
Working the first steps: the starting estimate 8 has variance 1, and the first measurement 10.4 also has variance 1, so K = 0.5 and the estimate becomes 8 + 0.5 × 2.4 = 9.2, with variance 0.5. The second measurement, 9.6, gets a gain of 0.5 / 1.5 = 0.333, giving 9.2 + 0.333 × 0.4 = 9.333. Each step the filter trusts itself a little more and the measurements a little less.
3D data and homogeneous transformations
Many self-driving cars carry a LIDAR sensor, which spins laser beams and measures how long their reflections take to return, producing a point cloud of hundreds of thousands of 3D points per second. Combining it with cameras requires converting coordinates between sensors, each of which has its own frame of reference. Homogeneous coordinates make this tidy: a point (x, y) is written (x, y, 1), so that rotation and translation together become a single matrix multiplication, and several transformations chain by multiplying their matrices.
For example, rotating by 90 degrees and then shifting by (1, 2) uses the matrix with rows (0, −1, 1), (1, 0, 2) and (0, 0, 1). Applied to the point (3, 0, 1), it gives (0 × 3 − 1 × 0 + 1, 1 × 3 + 0 + 2, 1) = (1, 5, 1), so the point lands at (1, 5). The same idea in 3D uses 4 × 4 matrices. Projecting a 3D point onto the camera image uses the camera's focal length and centre: with a focal length of 700 pixels and an image centre at x = 600, a point 2 metres to the right and 20 metres ahead appears at 700 × 2 / 20 + 600 = 670. Driving datasets supply these calibration matrices so that laser points can be drawn on camera images and 3D boxes projected into photos.
Cameras and LIDAR have complementary strengths. Cameras are cheap, see colour and texture, and can read signs and traffic lights, but they measure distance only indirectly and struggle in darkness and glare. LIDAR measures distance directly and precisely in any lighting, but it is more expensive, sees no colour and returns sparser data for objects far away. Some companies rely mainly on cameras and estimate depth with neural networks, while others combine both. Sensor fusion, merging the information from several sensors into one consistent picture, is only possible because the transformations between their frames of reference are known precisely, which is why calibration is checked so carefully.
Multi-task learning, 3D detection and bird's-eye views
A car needs several outputs from the same camera image, such as segmentation and depth. Training separate networks wastes computation, so multi-task learning shares one network for several tasks. The Multi-Task Attention Network (MTAN) uses a shared backbone and, for each task, small attention modules that learn which shared features matter for that task. The total loss adds the task losses, and balancing their weights is part of the design.
| Task | Input | Output | Method in the course |
|---|---|---|---|
| Road segmentation | Camera image | Road label per pixel | Fully convolutional network |
| 2D detection | Camera image | Boxes with classes | YOLO |
| Tracking | Detections over time | Boxes with lasting identities | Deep SORT |
| Multi-task perception | Camera image | Segmentation and depth | MTAN |
| 3D detection | LIDAR point cloud | 3D boxes with orientation | SFA3D |
| Bird's-eye view | Several camera images | Top-down map of the scene | UNetXST |
3D object detection finds objects as boxes in space, with position, size and heading. SFA3D, a fast single-stage method, first converts the point cloud into a top-down image whose pixels record height, intensity and density of points, then detects objects in that image with a convolutional network, which is much quicker than processing raw points. Finally, planning is easiest on a bird's-eye view, a map seen from above. UNetXST learns to transform several camera images into one top-down semantic map, using known camera geometry to guide a U-shaped encoder-decoder network. Together these tasks give the planner a picture of the road, the objects on it and where each is heading.
Check your understanding
- A road prediction has 7,200 correct road pixels, 800 false road pixels and 1,000 missed road pixels. Compute the IoU.
- Compute the IoU of boxes (0, 0)–(4, 4) and (2, 2)–(6, 6). Would non-maximum suppression with a threshold of 0.5 remove one?
- Continue the Kalman example from estimate 9.333 with variance 0.333 and a new measurement of 10.3. Give the gain and the new estimate.
- Apply the rotate-and-shift matrix from this chapter to the point (0, 2).
- Why does Deep SORT use appearance features as well as predicted motion?
- Explain why multi-task learning can be more efficient than separate networks, and one difficulty it introduces.
This course book is available to read online only at tech-academy.chrysolites.org.