How Team Signage optimized categories for street signs

[karansaini282]

• The first step was identifying the official street signs in the United States.

• The broad categories of signage were warning signs, temporary road signs, regulatory signs, mandatory movement signs, guide signs, services signs, pedestrian signs, bicycle signs.

• Images were downloaded using Google Images by writing a Selenium script to automatically download an input number of images for an input search. There were some duplicate images which were automatically deleted by Custom Vision on uploading.

• There was a slight imbalance in the number of images and hence warning signs and temporary road signs were clubbed into one.

• At the end of the last iteration each category had the number of images containing their label ranging from 50-150. Custom vision recommends a minimum of 30 images per category label.

• The model chosen on Custom Vision was "Logos". This made sense since street signs have distinct shapes, symbols and colors for different kinds of categories.

• The model was run for 12 iterations in total for a satisfactory performance. At the end of each iteration custom vision provides accuracy, recall and precision for the overall model as well as per category label.

• After each iteration we targeted the category labels with the worst performance. Possible solutions included downloading more images for that label, combing the label with another similar label or multiple labels for a single bounding box.

• The final performance metrics were 94.7% precision and 83.3% recall. Recall was the most difficult metric to optimize since a number a images had very small bounding boxes for street signs which were difficult to be identified up by the model.

[patwater]

Great stuff! Regarding performance metrics it'd be great to generate a confusion matrix to explicitly see the breakdown of false & true positives / false & true negatives

[vr00n]

@karansaini282 at our next check-in - i'd love for you to quickly go over your selenium script and how you use selenium for other extraction activities.

Thanks!

[karansaini282]

@patwater thanks for the tip! Confusion matrices are a great way of truly understanding where a model needs improvement.
@vr00n Absolutely, I will try my best to go over my extraction scripts and Selenium in general. I find it very useful to generate my own data in personal projects.