In this article I will demonstrate how Computer Vision can support the process of turnaround management in the aviation industry. Therefore I’ve developed a demo application and decided to share my thoughts I had during the development.
But first let’s clarify what turnaround management really is by learning from the experts:
Turnaround management, when applied to the aviation industry, refers to the physical process of preparing an aircraft for its next flight […]. The process of turnaround management is addressed when an aircraft is on the ground — from the time it arrives at a terminal gate, to the time it departs on its next scheduled flight.
In my simple words, turnaround management is everything that has to be done on the ground to prepare the aircraft for the next flight including boarding, baggage loading and fueling.
This process is crucial for every airline and should be optimized in every way possible to reduce downtimes, save costs and make customers satisfied with in time flights.
One of the main problems optimizing the process is the lack of actual knowledge about the current performance in the fleet. In this demo I show my way of using Computer Vision to extract this knowledge from camera streams at airports.
I am going to use different software products for different tasks:
- SAS Viya to develop an object detection model
- SAS Event Stream Processing to analyze the video in stream (also called SAS ESP)
- Python interfaces from SAS (SAS SWAT, SAS DLPy, SAS ESPPy) to communicate with the SAS software
- Open Source libraries like OpenCV, Numpy and matplotlib for analytical and visual tasks, e.g. polygon calculation and visualizing results
As you can see, this work is a combination of powerful SAS software and easy-to-use open source packages in Python. Why choose one if you can have both? 🙂
But now let’s get our hands dirty!
1. Define your business processes
Yes, everyone loves training computer vision models and visualize the results. But before starting your precious GPUs you have to decide what business processes you’re interest in and how they are defined.
I am by no means a turnaround expert so don’t expect too much from my definitions. You can define them as complex as you need them. For this demo I kept it simple, e.g. : If I have at least 2 people on the stairways I assume that the boarding has started.
Defining these business processes will support you in the next two parts.
2. Define the objects you want to detect and train a model
Now, given the process definitions it will be easy to create a list of objects you want to detect. In my case I decided to train the model for the following objects: aircraft, person, baggage truck, ramp loader, bus, fuel truck, tank hose, ground power, stairway and rolling stairway
Yes these are more objects than I have in my process definitions but I thought I could maybe use some of them later. 🙂
If you’re unlucky like me and don’t have access to training data you can try to find videos on YouTube. For my example I used this video and labeled it using CVAT.
Finally, yes, now is the time for your GPU power!
For object detection I used a standard Tiny YOLO V2 model but other object detection models are possible. The training process is straight-forward but you can have a look at it here.
3. Define Areas of Interest
With your model being able to detect relevant objects you should now go back to your process definitions and extract the relevant areas for your business process. For example you don’t want to count the guys lifting your heavy baggage into the aircraft as passengers boarding the aircraft.
Looking at my definitions, I’ve decided to create the following five Areas of Interest: aircraft area, baggage area, stairway 1, stairway 2, fueling area
Areas of Interest and their corresponding Task DescriptionsFor demo purposes I decided to manually define the Areas of Interest. However, you could of course define them dynamically given the objects you detect. This way the application would be more general, e.g. to fit other aircraft types or airports.
4. Create your Turnaround Management application
Now comes the fun part and we are putting all the pieces together to develop our Computer Vision aided Turnaround Management application.
As said in the beginning I’ve used SAS Event Stream Processing to create the image analysis process and afterwards connected this stream to SAS Viya to serve a simple SAS Visual Analytics dashboard showing a turnaround timeline.
Turnaround Image Analysis in SAS Event Stream Processing
The SAS ESP process is very standard at the beginning. We are loading and deploying or computer vision model and provide it to the scoring window. Additionally we are receiving the camera image and resize it to fit the needs of the YOLO v2 model.
The YOLO v2 model is applied to the images and we receive the object coordinates. These object coordinates are then fed into a calculation window which will apply our process definitions and produce the visualizations.
Because I am a big fan of programming in Python, I decided to code the business rules in Python too. This is not a problem because the calculation windows in ESP allow us to run Python code via SAS Micro Analytics Services. There are even “PythonHelper Windows” in the SAS ESPPy interface to SAS ESP where you only have to provide the location of your code making the whole process very simple.
The calculation window produces one line of data per frame, including the scored image and several timestamp variables. I’ve used OpenCV to display the scored frames and also coded a small and ugly “dashboard”. Having this dashboard next to our image allows us to check whether all processes have been identified correctly.
Analyzing the video stream of a Turnaround - Visualized output of the SAS Event Stream Processing PipelineTurnaround Dashboard in SAS Visual Analytics
Even though looking at the results gives us already a feeling of being done, this is usually not the case. Applying computer vision models to your images is useless if you don’t make use of the information they extracted from images.
Therefore I decided to use the SAS Cloud Analytic Services Adapter to push the results into my SAS Viya environment. This allows me to use the data directly for further analysis, e.g. creating nice reportings or even do more sophisticated stuff like predictive modelling or forecasting.
All I had to do is to add a transpose window at the end of my SAS ESP process that creates the nice and small 3-column table that you can see in the diagram.
The rest is simple dashboarding as you would expect it from professional reportig and analysis software.
I am not a great report designer but you can have a look at my example report here:
Streaming the analyzed data to SAS Visual Analytics for Live Dashboarding
On the left side I used a so called “Data -Driven Content Object” that links to a simple websocket connector written in JavaScript to grab the scored frames from my ESP process. (Yes you can connect to ESP via a lot of different ways!) The good thing about a Data-Driven Content Object is that you can not only receive data but also send data to it directly from VA. This would enable us to interactively change the connection to a different ESP process that observes another aircraft.
The left side uses the transposed data coming from the ESP CAS adapter to create a simple timeline chart. The report refreshes its data every second to accurately show the current process status.
With the data being available in SAS Viya for all kinds of analysis, I’d like to finish this article and would love to hear about your thoughts.
Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast
LinkedIn | GitHub | Medium.com
Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.
Find more tutorials on the SAS Users YouTube channel.
-
2080 views
-
1 like
-
-
