Using computer vision, virtual reality and streaming data to detect potholes
What if I told you that you could create a small town complete with potholes in virtual reality and then use it for computer vision model training? Would you believe it possible? The benefits of training in VR are many:
- Ability to rapidly control and change environmental factors like the change of seasons, rainfall, snow even time of day.
- Vastly lowers the cost of physical assets during ideation and prototyping.
- Data collection and training now takes seconds instead of days or months.
- Ability to model when access to the physical environment or situation isn't possible.
Pairing SAS Analytic for IoT with the Epic Games Unreal Engine and Microsoft's project AirSim the seeming impossible becomes possible. Check out this video to learn more:
Potholes are bad
The financial impact of potholes and other effects of under-maintained roadways has a real impact on the wallets of motorists across the U.S. Each year potholes are causing an estimated $3 billion in blown tires, busted axles, and other collateral damages.
Municipalities have developed some ways to prevent potholes from forming in city streets like crack sealing, road patching, and overlays, and these preventative efforts can help reduce damage to vehicles and costly roadway repairs. The main problem is that the same municipalities don't have ways to timely identify the areas that need intervention. This demo shows how visual detection technology can dramatically help cities keep road maintenance costs down and improve overall public safety by identifying and ranking potholes based on their condition so that repairs can be pre-emptively executed.
Usually, garbage trucks rumble round the city streets doing a simple job – providing and transporting collected trash. The idea is to convert these humble, single purpose trucks into a road maintenance information gathering hub. By fitting cameras at the front/back of garbage trucks and having an SAS Event Stream Processing engine running with a trained model to identify potholes. Information about the pothole like size and location can then be sent in real-time to a central hub or stored in a database and presented to repair crews in an orderly way. More traditional analytics can then be run there to decide which section of roads can be targeted to be patched first to minimize effect on traffic and maximize crew time utilization.
The solution
For this solution we will use a virtual environment to simulate a small town, due to the fact that it is not feasible to mount cameras on real trucks. An open source project called AirSim will be used. AirSim is a simulator for drones, cars and more, built on Unreal Engine . It is developed as an Unreal plugin that can simply be dropped into any Unreal environment.
For this demo we built a virtual small town which includes streets filled with potholes.
The AirSim plugin was added to this environment which allows for the capture of images from the cameras which are attached to the vehicles which traverse the town using a python API. Model training is done by simply replacing the real world components for the collection of images with our virtual stack. It all looks something like this:
A python API is used to control a vehicle which traverses the virtual environment. Attached to the vehicle are cameras which are used to collect images of the town's road conditions. These images are then used by SAS VDMML to train an object detection model. Once trained the model is deployed into SAS ESP to score new images as they are created. The simulator data also includes location information as well as the images. Using a combination of computer vision and location information we can create a database which contains current road conditions and plot them on a map using SAS Stream Viewer.
The output from the training exercise becomes an input to the ESP Build Process. ESP is where the real time images and data are processed to generate the dashboard. The demo conceptualizes the use of 4 separate garbage trucks. Data from truck number 1 is collected from the virtual town live using the virtual camera mounted to the truck. Trucks 2-4 represent data previously collected and stored in a data file. Each truck is represented on the map as a different color circle. The collected images will be inferenced using a model trained by SAS using DLPY.
The left box represents our world. An ESP edge server which contains the trained model inferences images in real time. When a pothole is detected the results are formatted along with the truck ID and geolocation data and sent over MQTT to the ESP server running in the cloud. The cloud server collates data from each virtual truck into the dashboard.
Conclusion
Visual detection technology is an effective way to detect potholes and help municipalities keep road repairs costs down while improving safety conditions. The benefits extend beyond road repairs, as preemptive intervention can also help shield local administrations from costly vehicle repair bills. Leveraging a virtual environment helps reduce costs and speeds up the model training process. It is a great first step in the model building process and can be augmented when possible with real world images when available.
