Due to the Corona crisis many governments decided that people should wear face masks when they’re in public areas such as public transporation, supermarkets, etc.   Closely related to my other story where I track social distancing rules this demo shows how Computer Vision can help us to identify whether people wear a mask or not.   In this example I train a Tiny YOLOv2 model to detect faces and masked faces on images. This model is then used in a SAS Event Stream Processing project to score frames from a video. Such applications could be interesting if applied to camera streams, e.g. coming from CCTV.   Obtaining Training Images We need a lot of annotated images to train Computer Vision models. Collecting these images might be easy but labeling them costs a lot of time.   I spent roughly two hours to find and label images of people wearing and not wearing masks using Google image search. I ended up having 300 labeled images of people wearing masks and 150 of people without masks.   That amount is far away from being the perfect size of a training data set but we can try to enhance it with data augmentation.   Data Augmentation In the context of simple image classification where we have one label per image it is easy to perform data augmentation. You can crop, rotate, shear, etc. your image as you like it — the label stays the same.   However, if you’re trying to train an object detection model, this doesn’t work anymore because your labels (coordinates) will change if you transform the image. To overcome this challenge we will not only transform our image but also the corresponding labels.   I wrote a small example function in Python that will perform horizontal flipping, HSV transformations (hue, saturation and value) as well as creating mosaic images consisting of 4 randomly sampled images. The labels are changed accordingly.   Using my augmentation function, I grew my training set from 450 images to ~9000 images. An example for a transformed training image looks like this: Example of an augmented image Training the Object Detection Model With the available training data the training process itself is straight forward and not really different to my other examples. I am again using the Python API from SAS called SAS DLPy. The training notebook can be found here.   First I load and display my training data: Displaying the training data Afterwards I split the data into a training and validation set and fire up the training on my GPU. I trained the model for 80 epochs and achieved a validation loss of ~2. Not perfect but enough for a small demo application. Training History Then the final model is exported in SAS binary format called “astore” which can easily be used in real time applications using SAS Event Stream Processing.   Applying the Model to Videos The advantage of using YOLO as an object detection model is its capability of running in real time, e.g. using videos coming from CCTV.   I developed a simple pipeline in SAS Event Stream Processing using the Python API SAS ESPPy that loads my trained model (bottom part) and receives video frames (top part). The frames are then scored in the last box. The notebook can be found here. SAS Event Stream Processing Pipeline The result of this pipeline is a scored frame with coordinates and classes for our detected objects. Using simple coding in OpenCV we can display and save the results in a file.   More useful applications could use the extracted objects to generate alarms or restrict access to people without masks.   Final Results I used my webcam to make a short video of myself wearing various different masks. The model is able to distinguish between masked and unmasked faces indicating the prediction with green or red bounding boxes.   Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast The whole project can be found in my GitHub repository. ... View more

In my last community article I shared some insights about Deep Learning with SAS Viya on GPU Accelerated Docker Containers. SAS Viya is the perfect choice for developing and scoring deep learning models as long as you are not dealing with applications that need real-time performance.   SAS Event Stream Processing for Streaming Analytics For real-time applications, such as analyzing video streams, you should turn to SAS Event Stream Processing. Deep Learning models developed in SAS Viya can be easily exported in the astore format. You can then integrate the astore model into your SAS Event Stream Processing process. This is the approach behind most of my Computer Vision demos. Have a look at these two small code examples that demonstrate the process: # Have a look at the save_to_astore function from SAS DLPy. # This function will export your model as an astore. # Assuming your trained model is called model model.save_to_astore("/path/") # In the next step we will use SAS ESPPy to load the astore model. # We use a Publisher to send the model information into our ESP # process and activate GPU processing with USEGPUESP. publisher.send('i,n,1,"USEGPUESP","1"\n') publisher.send('i,n,2,"ndevices","1"\n') publisher.send('i,n,3,"action","load"\n') publisher.send('i,n,4,"type","astore"\n') publisher.send('i,n,5,"reference","model.astore"\n') publisher.send('i,n,6,,\n') publisher.close()   SAS Event Stream Processing on Edge There are two different versions of SAS Event Stream Processing: SAS Event Stream Processing SAS Event Stream Processing on Edge The second option is the perfect choice if you plan to use SAS ESP with Docker containers. It doesn’t come with unnecessary user interfaces but focuses on programmatical development. I usually use the Python interface.   Note: We will also use this version for deployment on edge devices such as NVIDIA Jetsons, which will be explained in my next community article.   SAS Event Stream Processing on Edge with GPU accelerated containers If you take a look at the documentation of SAS ESP on Edge, you will see that Docker Deployments are officially supported. However, they currently do not support GPUs. A No-Go for scoring deep learning models in real-time.   To still support GPUs, we can create our own Docker Container. As described in my last article about SAS Viya, we are again using the NVIDIA Container Toolkit and a base image from NVIDIA that supports GPUs.   In addition to SAS Event Stream Processing on Edge we will install common Python packages that will help us developing our streaming applications. The most important one being SAS ESPPy, our Python interface to the SAS Event Stream Processing and OpenCV 4.4.0 for Computer Vision tasks.   Get your hands dirty If you have a license for SAS Event Stream Processing on Edge, an NVIDIA GPU and are interested in deep learning, you can take a look at my GitHub repository to see how to create an executable container in a few steps.   SAS Event Stream Processing Docker Container with GPU acceleration   Once your container is up and running, feel free to browse my other GitHub repository to see what kind of applications you could develop.   Final Remark Please note: This is my private work and not officially supported by SAS. Feel free to ask me any questions but don’t expect official support from SAS. Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast LinkedIn | GitHub  | Medium.com  ... View more

As a Datascientist at SAS, I have the opportunity to keep up with the latest technological developments in machine learning. Recently, this means especially the development of applications based on Deep Learning. You can have a look at some recent examples here. One of my better known examples for deep learning is the tracking of social distancing that I explained in this community article.     Today I’d like to share some insights what kind of infrastructure I’ve setup to support my daily work of developing deep learning prototypes.   Why Deep Learning? Customers want to analyze unstructured data The reason is simple: customers want to analyze more and more unstructured data such as images, sound and text in addition to classic, structured data. The use cases vary a lot but in most cases they are built on Deep Learning algorithms that achieve state-of-the-art results.   Powerful hardware for Deep Learning Neural networks, which form the basis of Deep Learning, are not a new development. However, one of the reasons why Deep Learning has been experiencing such a strong upswing for some time now is certainly the progress in the field of specialized hardware. Especially graphics cards with their many computing units (GPUs) form the basis for the development of state-of-the-art Deep Learning models.   Graphical Processing Units (GPUs) accelerate Deep Learning Strongly parrallizable tasks, such as those involved in the computation of Convolutional Neural Networks (CNNs), are no longer performed on CPUs, but are shifted to the many specialized computing cores of the GPU. Watch this funny video from NVIDIA, to get a rough understanding why GPUs are so much faster than CPUs:     The Deep Learning framework from SAS supports the calculation on NVIDIA graphics cards. This provides significant performance advantages during training and scoring of models. In tests I have achieved performance boosts by the factor 1000x or higher. Of course this strongly depends on your network architecture as well as what hardware you’re using.   Quick and Dirty Deep Learning Development with Docker Containers As a Datascientist in the presales department I develop many prototypes to show customers the magic of Deep Learning. For this reason I was looking for a way to have an executable development environment within seconds on which I can develop and run my prototypes. This environment should also be easy to setup and share with my colleagues.   I went through the process of creating a single docker container. This container contains all the components necessary to develop Deep Learning applications. It is based on the NVIDIA Container Toolkit, which allows the creation of GPU containers. Using one of NVIDIA’s base images, I installed a SAS Viya Programming Only and a Python environment including SAS provided Python APIs for Deep Learning development. Programming Only means that this container includes no visual interfaces, e.g. SAS Visual Analytics, because they are not needed for Deep Learning development. That way we are also reducing container image size and startup time a lot.   Get your hands dirty If you have a license for SAS Viya and are interested in deep learning, you can take a look at my GitHub repository to see how to create an executable container in a few steps.   SAS Viya Programming Only Docker Container with GPU acceleration   Once your container is up and running, feel free to browse my other GitHub repository to see what kind of applications you could develop. I’d also like to encourage you to have a look at our official examples in our Deep Learning Python API GitHub.   SAS and Real Time Analytics SAS Viya is the right tool to develop and score your deep learning models. But what about analyzing streaming data such as videos? SAS provides another solution for that: SAS Event Stream Processing and SAS Event Stream Processing on Edge (for edge devices like NVIDIA Jetsons). I’ve also developed containers for these tools and I’ll share my container setups for SAS ESP and SAS ESP on Edge in the following community articles.   Final Remark Please note: This is my private work and not officially supported by SAS. Feel free to ask me any questions but don’t expect official support from SAS (yet). Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast LinkedIn | GitHub  | Medium.com  ... View more

In this article I will demonstrate how Computer Vision can support the process of turnaround management in the aviation industry. Therefore I’ve developed a demo application and decided to share my thoughts I had during the development. But first let’s clarify what turnaround management really is by learning from the experts:   Turnaround management, when applied to the aviation industry, refers to the physical process of preparing an aircraft for its next flight […]. The process of turnaround management is addressed when an aircraft is on the ground — from the time it arrives at a terminal gate, to the time it departs on its next scheduled flight. -Collins Aerospace Airport Knowledge In my simple words, turnaround management is everything that has to be done on the ground to prepare the aircraft for the next flight including boarding, baggage loading and fueling. This process is crucial for every airline and should be optimized in every way possible to reduce downtimes, save costs and make customers satisfied with in time flights. One of the main problems optimizing the process is the lack of actual knowledge about the current performance in the fleet. In this demo I show my way of using Computer Vision to extract this knowledge from camera streams at airports. I am going to use different software products for different tasks: SAS Viya to develop an object detection model SAS Event Stream Processing to analyze the video in stream (also called SAS ESP) Python interfaces from SAS (SAS SWAT, SAS DLPy, SAS ESPPy) to communicate with the SAS software Open Source libraries like OpenCV, Numpy and matplotlib for analytical and visual tasks, e.g. polygon calculation and visualizing results As you can see, this work is a combination of powerful SAS software and easy-to-use open source packages in Python. Why choose one if you can have both? 🙂 But now let’s get our hands dirty!   1. Define your business processes Yes, everyone loves training computer vision models and visualize the results. But before starting your precious GPUs you have to decide what business processes you’re interest in and how they are defined. I am by no means a turnaround expert so don’t expect too much from my definitions. You can define them as complex as you need them. For this demo I kept it simple, e.g. : If I have at least 2 people on the stairways I assume that the boarding has started. Aircraft Turnaround Task Definitions Defining these business processes will support you in the next two parts.   2. Define the objects you want to detect and train a model Now, given the process definitions it will be easy to create a list of objects you want to detect. In my case I decided to train the model for the following objects: aircraft, person, baggage truck, ramp loader, bus, fuel truck, tank hose, ground power, stairway and rolling stairway Yes these are more objects than I have in my process definitions but I thought I could maybe use some of them later. 🙂 If you’re unlucky like me and don’t have access to training data you can try to find videos on YouTube. For my example I used this video and labeled it using CVAT.   Finally, yes, now is the time for your GPU power! For object detection I used a standard Tiny YOLO V2 model but other object detection models are possible. The training process is straight-forward but you can have a look at it here.   3. Define Areas of Interest With your model being able to detect relevant objects you should now go back to your process definitions and extract the relevant areas for your business process. For example you don’t want to count the guys lifting your heavy baggage into the aircraft as passengers boarding the aircraft. Looking at my definitions, I’ve decided to create the following five Areas of Interest: aircraft area, baggage area, stairway 1, stairway 2, fueling area Areas of Interest and their corresponding Task Descriptions For demo purposes I decided to manually define the Areas of Interest. However, you could of course define them dynamically given the objects you detect. This way the application would be more general, e.g. to fit other aircraft types or airports.   4. Create your Turnaround Management application Now comes the fun part and we are putting all the pieces together to develop our Computer Vision aided Turnaround Management application. As said in the beginning I’ve used SAS Event Stream Processing to create the image analysis process and afterwards connected this stream to SAS Viya to serve a simple SAS Visual Analytics dashboard showing a turnaround timeline. From Streaming Data to Live Dashboarding Turnaround Image Analysis in SAS Event Stream Processing The SAS ESP process is very standard at the beginning. We are loading and deploying or computer vision model and provide it to the scoring window. Additionally we are receiving the camera image and resize it to fit the needs of the YOLO v2 model. The YOLO v2 model is applied to the images and we receive the object coordinates. These object coordinates are then fed into a calculation window which will apply our process definitions and produce the visualizations.   Because I am a big fan of programming in Python, I decided to code the business rules in Python too. This is not a problem because the calculation windows in ESP allow us to run Python code via SAS Micro Analytics Services. There are even “PythonHelper Windows” in the SAS ESPPy interface to SAS ESP where you only have to provide the location of your code making the whole process very simple.   The calculation window produces one line of data per frame, including the scored image and several timestamp variables. I’ve used OpenCV to display the scored frames and also coded a small and ugly “dashboard”. Having this dashboard next to our image allows us to check whether all processes have been identified correctly. Analyzing the video stream of a Turnaround - Visualized output of the SAS Event Stream Processing Pipeline Turnaround Dashboard in SAS Visual Analytics Even though looking at the results gives us already a feeling of being done, this is usually not the case. Applying computer vision models to your images is useless if you don’t make use of the information they extracted from images. Therefore I decided to use the SAS Cloud Analytic Services Adapter to push the results into my SAS Viya environment. This allows me to use the data directly for further analysis, e.g. creating nice reportings or even do more sophisticated stuff like predictive modelling or forecasting. All I had to do is to add a transpose window at the end of my SAS ESP process that creates the nice and small 3-column table that you can see in the diagram. The rest is simple dashboarding as you would expect it from professional reportig and analysis software. I am not a great report designer but you can have a look at my example report here: Streaming the analyzed data to SAS Visual Analytics for Live Dashboarding On the left side I used a so called “Data -Driven Content Object” that links to a simple websocket connector written in JavaScript to grab the scored frames from my ESP process. (Yes you can connect to ESP via a lot of different ways!) The good thing about a Data-Driven Content Object is that you can not only receive data but also send data to it directly from VA. This would enable us to interactively change the connection to a different ESP process that observes another aircraft. The left side uses the transposed data coming from the ESP CAS adapter to create a simple timeline chart. The report refreshes its data every second to accurately show the current process status. With the data being available in SAS Viya for all kinds of analysis, I’d like to finish this article and would love to hear about your thoughts.   Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast LinkedIn | GitHub | Medium.com  ... View more

Social media platforms such as Facebook, WhatsApp and WeChat are wonderful ways to communicate with people all over the world. Unfortunately these platforms are also used by criminals which is the reason why crime fighters want to investigate the contents on these platforms. While these platforms have stored a lot of metadata which is more or less available (especially for government organizations), the most interesting content is of course the one that is created by the users. This content usually consists of unstructured data such as text, speech, videos and images. In this article I am focusing on the latter, images, but the approach itself can be used for all kinds of unstructured data.   Our task: Enabling efficient ways of investigating image data Now, imagine being part of an investigation group of the police that is responsible to fight weapon crimes, right-wing or terroristic organizations. You have scraped data from various channels, e.g. Twitter, Facebook, WeChat & Co. and now you’re sitting on a huge pile of images not knowing where to start looking. Manually investigating these data sources is impossible, due to their size and variety. Our goal is to provide an efficient way of filtering this data, allowing investigation officers to focus on the relevant images and their sources. This not only saves manual effort but also enables officials to better protect the law.   The idea: Image classification to create additional metadata The idea is simple. We will train an image classification model that is able to classify our images into various categories. So to speak, we generate additional metadata for our images. This classification is then combined with other data we already have such as source, date, location, etc. Finally we want to create an intuitive dashboard that enables investigation officers to quickly identify potentially interesting content and skip through all the nice and funny images of cats. 🙂 As soon as we have the trained model available, the workflow is the following: Metadata coming from new Images is directly fed into the analysis environment such as SAS Viya. Images are scored using our computer vision model Classifications and Metadata are combined The combined data from step 3 is then fed into a visual report allowing crime investigators to perform in-depth analysis. Getting our hands dirty: Training the image classification model I am using the deep learning capabilities of SAS Viya to develop the image classification model. Of course you can develop the model using the SAS programming language but I am more comfortable using Python and Jupyter Notebooks. Luckily there are powerful Python-APIs that allow you to interact with your SAS environment. I am using the following APIs SAS SWAT (communicate with SAS Viya) SAS DLPy (special deep learning API for SAS) In this Jupyter Notebook you can see how I use transfer-learning to train a ResNet-50 image classification model that afterwards can detect different weapons. The model was pretrained on the ImageNet dataset which contains images from 1000 classes and can be downloaded here. There is also a second example where I apply the same process to images from hate symbols such as ISIS flags or swastikas. The data used for training comes from two different sources: Kaggle Weapon Dataset Hate symbols that I scraped from Google Image search by hand Here is an example how my training data looks like for the weapon detection: The Jupyter Notebook comes with a lot of comments that explain in-depth what I am doing. Therefore I am not going into the details here to keep the article as short as possible. After training the model, I evaluated its performance using a confusion matrix. As you can see the predictions are quite accurate with an accuracy of 99%. This seems a little bit too high for me and might be due to the training data being really nice to us. A lot of images have a nice white background which you cannot expect from real social media images. 🙂 Finished? No! A nice model is useless for our investigation officers. A lot of articles about Computer Vision end when the model was trained and successfully scored some new data. However if you want your efforts to be useful you also have to think about how to put your models into the bigger picture. While SAS has advanced technology for this, e.g. the SAS Model Manager, I am not going into the details of production-ready applications here. Instead, I’d like to demonstrate how a simple dashboard that uses our model’s predictions could look like. The dashboard was built with standard functionality of SAS Visual Analytics. Example 1: Investigate Images of Weapons coming from Social Media Example 2: Detect Hate Symbols on Images coming from Social Media The only interesting part is the “Data-Driven-Content Object” that is responsible for displaying the selected images which you can see on the right side of the reports. This object dynamically receives the data selection from the other report elements. Behind the scenes it is using a very simple JavaScript code to retrieve and display the selected images. As you can see in the videos our model acts as a supplier of additional metadata. It allows us to use the filter in the top-left corner to select the image categories we are interested in.   This use case does not only apply to crime investigation but is much more general. Even though this article focused on identifying crime there are of course many other interesting use cases. Another idea could be that you want to identify your brand’s logo or products in social media posts. This way you could for example promote authentic user postings that highlight advantages of your products or you can try to give additional support if it is a customer complaining.   Do you have other use case ideas for this approach? I’d like to hear about them in the comments!   Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast LinkedIn | GitHub | Medium.com  ... View more

Due to the recent pandemic, several governments have decided to implement restrictions for social distance. While most people follow these guidelines, there are still people who ignore them for various reasons. The overall goal of this project is to identify objects (i.e., people) who are following social distancing guidelines and those who are not. This is accomplished in the following way: Transform images and object coordinates into a two-dimensional map if a homography matrix was provided Calculate real world distances between detected objects if a homography matrix was provided. Otherwise use distances from the image directly. Detect crowds that exceed a specified parameter Visualize the results on the camera image Object Detection The first step in the process is to detect people in images. A Tiny Yolo V2 model was used for this task, but any detection model would work. The decision to use Tiny Yolo V2 was mainly driven because of its ease to use in SAS. Transforming the Image and Calculating Distance The second step is to calculate the distances between all objects. While this may sound simple, it is more complicated if you think about it. Normally a camera does not provide a top view. Instead it is at some angle which leads to certain perspective. This perspective can be very important when you calculate real world distances in images. The following three pictures illustrate the transformation process. The second image shows the results after transforming the image using a Homography Matrix. A rectangle was able to be formed, but the distances are incorrect due to the camera's perspective. The third image show the results after a transformation matrix was applied. A distance of 356 pixels on the image shrinks to 149 pixels while a distance of 62 pixels in the other direction are 83 pixels in reality. Identifying Crowds Now that we have detected people and transformed the distances between them we want to determine whether we have crowds in our image. In this demonstration, KD-Trees from Scipy was used to look up nearest neighbors given a detected person and a maximum radius. The following functions were used: cKDTree to efficiently calculate nearest neighbors. query_ball_tree to query the KDTree with a person and a given maxium radius. Streaming Process The streaming process uses SAS Event Stream Processing (ESP) and can connect to any video data source. ESP lets you define the process either graphically or programmatically via a Python Interface. The following diagram illustrates the streaming process: The top two boxes load the trained Tiny YOLO V2 model and provide the model to the scoring window. The scoring window receives images that are resized to appropriate dimensions (416x416 pixels in this case). The scoring window provides the detected persons and their corresponding x, y, width, and height values. The last box utilizes the Python inside SAS Event Stream Processing to transform the coordinates given the homography matrix. Additionally, it uses Scipy to perform crowd detection. Ressources & Results For a more detailed look (incl. code) at Tracking Social Distancing Using Computer Vision, refer to this GitHub page. And now let's view the results! The left side shows the original camera image that was annotated by our streaming process. On the right you can see the top view produced by our homography matrix. Green colors show long distances, orange and red indicate small distances. Last but not least blue squares are drawn around crowds of three or more people.   Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast LinkedIn | GitHub | Medium.com  ... View more