Building an Image Processing Pipeline in SAS ESP (without pre-trained deep learning models)

[#1]

Modern computer vision solutions often rely on trained deep learning models - but what if you need a fast, explainable, no-training-required approach for object detection?

In this post, I’ll introduce you to a powerful SAS Event Stream Processing (ESP) capability that demonstrates exactly that: a real-time image processing pipeline that detects objects obstructing a fire exit - using only built-in algorithms and a scoring API.

Here are some key points from this post:

1. No model training is required. Just use available image processing techniques.
2. The intelligence is not just in models it’s also in the pipeline design.
3. ESP has a real-time scoring API that supports synchronous request-response workflows.
4. Use this production-ready pipeline design for your next CV object detection project.

Our use case is simple but impactful: Think of this as a practical template for detecting objects in any fixed-camera scenario such as for monitoring evacuation routes, ATMs, workplace safety and restricted areas.

In this scenario, a fixed camera monitors a fire exit. If an object (person, bag, bin) blocks the exit, the system detects it and returns annotated results for downstream alerting.

The ESP pipeline uses the following baseline captured static image showing the unobstructed fire exit that does not violate regulations. This image is processed together with a new incoming event image (in the project's w_background_subtract window) to detect objects blocking the exit.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

An event (jpeg image) captured by the fixed camera is read into the project's w_source window. If something is blocking the exit such as a wastepaper bin shown below, this is considered a safety violation.

Using ESP windows in the pipeline, the pixels from the baseline static image are subtracted from those in the captured event image, leaving only any object such as this bin that obstructs the fire exit.

Bounding boxes are then added to any remaining obstructions that are detected by ESP, and the resulting image can be used for downstream alerting and analysis.

Many clever image processing techniques are used in this example to refine and improve detection accuracy. The ESP project windows execute the built-in SAS Event Stream image processing algorithms (for example, background subtraction, grayscale conversion, contrast enhancement, Gaussian blurring, adaptive binarization, and Connected Component Labeling).

This example is particularly interesting since unlike typical computer vision use cases that deploy a pre-trained model, this project requires no prior model training or separate ONNX (Open Neural Network Exchange) deep learning model. Just to be clear, ESP does support use of deep learning computer vision models for streaming analytics cases. These are commonly used in manufacturing and workplace safety monitoring situations.

The SAS Event Stream Processing Scoring API also works with other data types. The scoring API is great for testing an ESP project before putting it into production. You will see it’s usefulness for testing in several upcoming screen captures. From the ESP test mode, a JPEG frame is sent in an HTTP POST request directly to a running ESP project and a result is returned.

Project Package

When working on a new project, always start with a project package. This keeps all your necessary parts (files, models, metadata…) in a standard location where they can easily be accessed and where new “parts” can be added by uploading. A project package can also be exported and shared with others. Another benefit of using a project package is that it can be run immediately when it is subsequently imported into an ESP environment, just as with the available example projects.

Depending on the type of project you are working on, you would usually first build the source window and schema, test access to data using a file/socket connector to a test csv file and then add more streaming analysis windows to fill out your project. When you enter test mode, ESP starts a container in the Kubernetes environment for your project to run in. You then view results for the windows you are interested in.

You may not have been aware of this, but you can select any valid combination of source and output windows in the scoring API properties that are at the project level for use in test mode! In this example you will see how easy it is to use the API for a computer vision object detection model.

Here is the Github location of the project used in this post that explains the specific purpose of each window in the pipeline.

Let’s take a tour of this project that I have already opened in ESP. First, notice the project package layout with it’s folders for storing the analytical model, test files and image data. The static.jpg file under test_files is the baseline image used to subtract pixels from an event image.

ESP Scoring API properties

At the project level, scroll down to the Scoring API section. In the screen below, notice this option where you select the source and output windows that you want to test. The w_source window is where a JPEG frame is sent in an HTTP POST request to a running ESP project. After all image processing, object detection and bounding boxes have been applied, the selected w_annotate output window is where the resulting annotated image is created. These are the only two project windows required to set up the scoring API.

If everything is set up correctly, the project can be saved and now run in test mode.

When running a project in test mode, an ESP Server starts and once it initializes successfully, waits for input. I select the “Scoring API” tab in the middle of the screen and navigate to the project package folder having the test jpg file. Then I click “Send request”.

The processed image result and bounding box are displayed. I can save this output to another project folder.

I can test different image files, modify the behavior of windows as needed and when satisfied with the results, prepare the project for production deployment. Since this use case requires a fixed camera position and the baseline static image for pixel subtraction, any image files must also have that same baseline static image for valid subtraction to occur. So if an image of a skier in a snow capped mountain vista was tested in this project, no valid result would be created since the test image does not match the required static background.

 

In ESP Studio, I found it useful to modify the project scoring API settings and re-run the tests selecting different output windows. This helped me to understand how the image processing steps change the results throughout the pipeline.

Here are some of the intermediate results I captured from other windows after the base image was subtracted from the event jpeg file.

This scoring result is from a window where grayscale conversion and gaussian blur techniques have been applied to the subtracted image.

Below, adaptive thresholding is then applied to set the foreground pixels to white and the background pixels to black as part of enhancing the ESP pipeline’s object detection process. Notice how this step improves the result by identifying more crisp and distinct edges around the bin object.

Next, the screen capture below shows the amount of exit door overlap that is detected.

And once again, here is the fully processed violation image rendered by the final w_annotate window.

Be sure to keep the Scoring API and object detection capabilities in mind. You never know when they might improve your productivity when designing your next ESP project! 

Thanks for reading, and may all your fire exits be clear of obstructions!

Find more articles from SAS Global Enablement and Learning here.