Tracking Social Distancing Using Computer Vision

[mentos05]

Due to the recent pandemic, several governments have decided to implement restrictions for social distance. While most people follow these guidelines, there are still people who ignore them for various reasons.

The overall goal of this project is to identify objects (i.e., people) who are following social distancing guidelines and those who are not. This is accomplished in the following way:

• Transform images and object coordinates into a two-dimensional map if a homography matrix was provided
• Calculate real world distances between detected objects if a homography matrix was provided. Otherwise use distances from the image directly.
• Detect crowds that exceed a specified parameter
• Visualize the results on the camera image

Object Detection

The first step in the process is to detect people in images. A Tiny Yolo V2 model was used for this task, but any detection model would work. The decision to use Tiny Yolo V2 was mainly driven because of its ease to use in SAS.

Transforming the Image and Calculating Distance

The second step is to calculate the distances between all objects. While this may sound simple, it is more complicated if you think about it. Normally a camera does not provide a top view. Instead it is at some angle which leads to certain perspective. This perspective can be very important when you calculate real world distances in images. The following three pictures illustrate the transformation process.

The second image shows the results after transforming the image using a Homography Matrix. A rectangle was able to be formed, but the distances are incorrect due to the camera's perspective. The third image show the results after a transformation matrix was applied. A distance of 356 pixels on the image shrinks to 149 pixels while a distance of 62 pixels in the other direction are 83 pixels in reality.

Identifying Crowds

Now that we have detected people and transformed the distances between them we want to determine whether we have crowds in our image. In this demonstration, KD-Trees from Scipy was used to look up nearest neighbors given a detected person and a maximum radius. The following functions were used:

• cKDTree to efficiently calculate nearest neighbors.
• query_ball_tree to query the KDTree with a person and a given maxium radius.

Streaming Process

The streaming process uses SAS Event Stream Processing (ESP) and can connect to any video data source. ESP lets you define the process either graphically or programmatically via a Python Interface.

The following diagram illustrates the streaming process:

The top two boxes load the trained Tiny YOLO V2 model and provide the model to the scoring window. The scoring window receives images that are resized to appropriate dimensions (416x416 pixels in this case). The scoring window provides the detected persons and their corresponding x, y, width, and height values.

The last box utilizes the Python inside SAS Event Stream Processing to transform the coordinates given the homography matrix. Additionally, it uses Scipy to perform crowd detection.

Ressources & Results

For a more detailed look (incl. code) at Tracking Social Distancing Using Computer Vision, refer to this GitHub page.

And now let's view the results!

The left side shows the original camera image that was annotated by our streaming process. On the right you can see the top view produced by our homography matrix. Green colors show long distances, orange and red indicate small distances. Last but not least blue squares are drawn around crowds of three or more people.

Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast

LinkedIn | GitHub | Medium.com 

[Patrick]

That's very cool and very scary at the same time. Add facial recognition and weak privacy laws to it and 1984 becomes reality.

Or even just racial or social crowd profiling... I guess the privacy discussion about pictures, security cameras and its uses and data storage becomes more and more urgent. Who "owns" a picture of me? And what are you allowed to do with it? Hoping for an amendment to GDPR and the like!

[mentos05]

Thank you Patrick for your comment and I agree with you that one can imagine scary scenarios using Computer Vision technology.

Therefore, we as a society, have to make sure that we use such technology for good things - not bad things.

Speaking for this particular demo, the model used is a Tiny YOLO V2 network that is not able to recognize entities. It can only detect people in general and there is no plan from my side to enrich it with face recognition.

Thinking further, I would say that such systems can also help to ensure data privacy. After analyzing the video and calculating key figures (number of rule violations, people-density-index, etc.), we could immediately delete the video as we are usually not interested in the video itself but its condensed information that was extracted by computer vision.

Best Regards

Michael

[Patrick]

Thank you for your response @mentos05 and I have no doubt that SAS is doing the right thing for the right reasons. Having said that: Your post made me realize that the PII data question is of a much more complex nature than I've been aware of so far. My current question is: Is a picture/video of me PII or not? And is it already PII as such or only if "someone" publishes the picture or uses facial recognition or the like on it. And then taking GDPR regulations: Does this now mean that I could request from all persons/companies/agencies operating security cameras in Europe to delete me from any pictures/films they've taken from me anywhere I've been in Europe? And where is the list of these cameras and its owners? Feels like tons of unresolved practical and ethical questions.