Traffic Sign Detection and Classification using Deep Learning

Duration of the Thesis: 6 months
Completion: December 2016
Supervisor: Feng Liu, Volvo Car Corporation
Examiner: Prof. Dr.-Ing. Norbert Haala

In the context of driver assistance systems as well as autonomous driving, it is important that the vehicle can recognise traffic signs. Since the meaning of traffic signs might be alternated by supplementary signs as the one in figure 1, it is important to detect those as well. A lot of work has already been done to recognise the main traffic signs but not for supplementary signs. Therefore, this thesis focuses on those.

The detection and classification for both main and supplementary signs is tackled by applying the Single Shot MultiBox Detection network, short SSD, which can be seen in figure 2. This convolutional neural network has been published in March 2016 and is set up on a computer with Ubuntu and Caffe as framework for Deep Learning.

Different approaches for the training of the network have been tested for the supplementary signs as well as for the main signs. Thereby, images of the German Traffic Sign Benchmark are used for the main signs. One of those images can be seen in figure 3.

Since no public database for supplementary signs is available, supplementary signs rendered by Volvo were used. Those are merged with images from Google Street View leading to results as in figure 4. Next to those, a few supplementary signs that could be extracted from the German Traffic Sign Benchmark are included as well. One such sample can be seen in figure 5. The different classes of supplementary signs are shown in figure 6.

Figure 4: Example for a rendered supplementary sign sample
 

Figure 5: Example for a supplementary sign extracted from the German Traffic Sign Benchmark
 

The results show that the network struggles with objects that appear very small in the image as it is the case for traffic signs. The results can be improved with a sliding window approach but it is recommended to use other methods for the recognition of main signs. Figure 7 shows a image from the German Traffic Sign Benchmark and the corresponding detection. For the supplementary signs only the area around a main sign is used as input to the network. This leads to supplementary signs appearing large in the input. It as well is guaranteed that the corresponding main sign is available. A multiclass network is trained that is capable of distinguishing between the 9 different classes. Further a binary network is trained that can be used to create databases of supplementary signs by extracting them from images. For that purpose the network for supplementary signs can be used in combination with any good method for recognising main signs. Such a database allows future methods to be trained and tested on the same dataset. It then is possible to compare different methods with each other and to train on more samples and classes. Regarding the performance of the network on the supplementary signs, for both the multiclass and the binary case, the network can be recommended for that purpose. The results can be found in the thesis report.