Let's talk about  how to use two of the products found in SAS Analytics for IoT to train and build a demographic detector which may be used in a retail setting.   SAS Visual Data Mining and Machine Learning SAS Event Stream Processing These two products provide the tools needed when training, testing and running computer vision models. This article  will cover the following topics: Define the use case for our retail example Gathering and displaying demographic data, using object detection and image recognition Learn the basics of computer vision Learn how to train CV models using the SAS Deep Learning Python (DLPY) package How to deploy the trained CV models using SAS ESP How to present the demographic data effectively using a Grafana dashboard Use Case As the proprietor of a retail establishment, envision gaining a better understanding of your customers. Imagine gaining insights into the emotional engagement of customers as they explore your store, knowing the demographic preferences for specific store aisles, distinguishing between male and female customer attractions. Wouldn't that be invaluable? Using computer vision and streaming analytics it is possible to do this in real-time. All that's required is a camera connected through RTSP, complemented by an object detection model pinpointing human faces, alongside image recognition models discerning additional details. The outcomes are gathered and channeled into a dashboard, enabling real-time analysis. Gathering and displaying demographic data: a retail example   Using two products that are embedded in the SAS Analytics for IoT solution, SAS Visual Data Mining and Machine Learning (VDMML) and SAS Event Stream Processing (ESP), let's explain how to train and deploy various demographic analytic models to create a single solution that connects directly to an RTSP connected security camera. Through this connection, processing incoming images using pre-training analytical models produce demographic statistics. The following models will need to be trained for this use case: Face Detection - Process incoming images and detect human faces in the video stream Age Classification - Determine approximate age grouped by category Gender Classification - Determine male or female Emotion Classification - Group by these classes: Happy, Neutral, Sad, Fear, Surprise, Angry The example Grafana dashboard looks as follows:   Dashboard   There are various graphs displaying demographic statistics for the last 100 observations as well as a live video stream which shows the current face that has been detected via the RTSP stream. The image has been augmented to show the current results from the scoring algorithms. First off, let's talk computer vision   Computer vision (CV) techniques provide the ability to acquire, process and analyze incoming images. This analysis produces numerical results in the form of predictions based on the classes we define. In this example, we need to create four separate CV models. First, we need an object detection model which will not only give us the probability there is a human face in the incoming image, but it will also give us the coordinates of that face in the image, in the form of a bounding box. Using the box coordinates we can then crop just the face from the incoming image and send that data to the next analytical model. Consider this example:   Object Detection   Here I’m using a YOLO (You Only Look Once) object detection model to find all the human faces in the incoming video stream. In the case of multiple faces being detected, multiple cropped faces are passed to the downstream models as separate events. Next, we train our models   Before I can build an application that uses analytical models to predict outcomes, we need to train them. The training process for CV involves classifying images and separating these images into datasets that can then be fed into a machine learning model such as ResNet50, VGG16, Darknet etc. This stage of the process is completed using the SAS Deep Learning Python (DLPy) package which provides the high-level Python APIs to the deep learning methods in SAS Visual Data Mining and Machine Learning (VDMML).   Model Training   As the previous diagram illustrates, four separate datasets were created to support model training. Because I needed to tweak each dataset to get the best possible output, each image dataset supports the training of one model. Face detection was trained using a Yolo V2 architecture while age, gender and emotions were trained using a ResNet50 architecture. For example, when training the gender model, image data is loaded into SAS Viya and VDMML using DLPy. Deep learning algorithms are then invoked as each image is processed to create a portable analytics file called an ASTORE file. VDMML is GPU-enabled so that training times are greatly improved. A typical training exercise contains these steps: Setup libraries and launch CAS Load and explore the training data Prepare the data for modeling Specify the model architecture, configure model parameters and import pre-trained weights Fit the image detection and classification model Evaluate the newly created image classification model Visualize model results Save model as ASTORE for deployment Training   Model training is optional since the pre-trained .astore files have been provided. However, contained in the training directory of this repository are the Jupyter notebooks which were used during the demographic model training. Please refer to these notebooks in order to learn how to train your own computer vision analytic models. Please refer to the following for details:   Age training example Emotions training example Gender training example It is recommended that no less than 1000 images be used for each class you intend to train and more images are always better. For example, each age group or emotion class should contain no less than 1000 images. These images are also subdivided into two datasets. The first is for training and the second is a testing dataset. Therefore, an additional 200 images for testing is also recommended. Image size should be no less than 224 by 224 pixels. Image databases such as Open Image are a great place to start. For more information about the DLPy high-level Python APIs which allows you to build deep learning models please see the following examples:   Image Classification Object Detection Now use streaming analytics   Streaming analytics is defined as the ability to constantly calculate statistical analytics on an incoming stream of data. In our case, that stream of data is the images coming from the camera. SAS Event Stream Processing (ESP), which is part of the SAS Analytics for IoT solution, provides the ability to deploy our newly trained analytical models, in the form of ASTORE files, at the edge. With ESP you can ingest, filter and transform your data in stream. I like to think of it as enabling business rules to gain extra business value. For example, let's say your company wanted to track how many happy women between the ages of 10 and 30 walked by the frozen food section on Tuesday when you were running a sale, verses a normal day when the sale is not running. ESP gives you the capability to make that happen. There is a saying here at SAS, "Without deployment, analytics is only a science experiment." This diagram illustrates an overview of this project deployment architecture.   Architecture   Here we can see the flow of information through the system and highlight some key points:   ESP is built for speed. Although, there are many methods of ingesting data into ESP (REST, MQTT, MQ), to make this superfast I used a RTSP connector which allows me to directly connect ESP to the incoming video stream from the camera. I also took advantage of ESP's multithreaded capability by scoring age, gender and emotion simultaneously each in its own thread. ESP integrates with open source. You can easily call python scripts in stream from the ESP model flow. This allows further integration with other open source packages such as OpenCV. Using Python and OpenCV, the images were cropped, resized, reshaped and colors were manipulated. Anything is possible. Retention is amazing. Retention defines a group of events over a time or event count. Instead of doing analytics on each image individually, you can now take a group of images and create new event data – like the number of men over the last hour or total number of kids today. It is very powerful. ESP includes a powerful graphical development platform. Using ESP Studio models such as these may be created without any coding. For example, ingesting images via the RTSP connector is as easy as dragging a window onto the canvas and filling out an RTSP specifications form. ESP Grafana plugin seamlessly combines Grafana's intuitive dashboarding with ESP's versatility, offering real-time insights and dynamic visualization. ESP Project Detail The ESP project is best described in phases. Project package contents Data ingestion Analytical scoring OpenCV integration using python Grafana plugin integration ESP Project   As you can see the model is very readable and intuitive. Data flows in from the top blue box which represents our connection to the video stream. From there the image is resized and passed to our first algorithm which detects human faces. Using the ESP python window openCV is used to crop faces directly from the ESP server without using MAS or external python libraries. The cropped face image is then passed to our three demographic computer vision models. Model results are used to annotate the image in real time as well as calculate totals for the last 100 events coming into the project. These totals are then sent to Grafana and our dashboard is created. Project Package   Building your own demographic detector involves creating an ESP project package which includes the following trained models. Age.astore - CNN model which detects age Emotion.astore - CNN model which detects emotions Gender.astore - CNN model which determines gender faceDetection.Tiny-Yolo2.416.v1.astore - Object detection model which locates human faces in images A project package is useful for organizing related content within a single package. Once downloaded please include them in your project package as follows. It is best practice to create a project package and place your analytical models in the analytics directory. The model reader window will use this file location to load the model. Once the location of the ASTORE is known the XML is very straight forward. An ESP environment variable is used to define the location.   Project Package Model Reader Window   As mentioned above the model reader window references the models stored in the project package standard locations using an ESP environment variable. This allows for better organization and standardization. For the age astore model the reader window would be defined as follows:   <window-model-reader name="readagemodel" pubsub="true" model-type="astore"> <description><![CDATA[Read model which determines age]]></description> <parameters> <properties> <property name="reference"><![CDATA[@ESP_PROJECT_HOME@/analytics/Age.astore]]></property> <property name="usegpuesp"><![CDATA[1]]></property> <property name="NDEVICES"><![CDATA[1]]></property> <property name="DEVICE0"><![CDATA[0]]></property> </properties> </parameters> </window-model-reader>     This is also where you specify GPU usage for this model. If usegpuesp is set to 1, ESP will use the gpu when scoring this ASTORE. ESP Video Capture Connector   The SAS ESP video capture connector is a tool that reads an input video stream and publishes each extracted image frame into a Source window as a unique event that contains a binary blob field. It is one of the many connectors and adapters that extend SAS functionality. Using ESP Studio configure a Source window publisher as follows:   Connector Configuration   Analytical Scoring   ESP is configured to call 4 ASTORE models. The first streaming analytics model performs real-time object detection, where the object is defined as a human face. The output is mapped to the following schema. nObjects defines how many faces were found and the remaining schema entries define where in the image the object was located. Sample of object detection schema:   Partial Schema   These values will later be used to draw bounding boxes and create new cropped images of the face which was detected. The face image is then passed to the image classification models to determine mood, gender and age of the person. After all the models are run, we have 3 important strings in our schema that represent the results of our efforts.   Results   The counts assigned to these labels are aggregated later in the project to generate totals, which are then visualized as graphs on the Grafana dashboard.   OpenCV Using the ESP Python Window   ESP now contains a Python window that can be used to call python directly. Python window empowers you to craft functions within Python programs, generating SAS Event Stream Processing events. The Python environment includes most commonly used packages including Numpy and OpenCV. This project leverages these packages to manipulate images in the following ways: Cropping Annotating Reshaping Converting to greyscale Drawing bounding boxes Images are passed to the Python window as BLOBs. No B64 encoding is necessary. This improves performance. A handy Python code editor is also included to make things easy.   Code Editor As with the Lua window a function called create is called and passed the input schema in a variable called data. A function that uses openCV call crops_draw_bboxes() is called to do just that. Crop faces out of the main image creating smaller images that can be passed to the demographic ASTOREs for scoring. Grafana Plugin Setup   The new SAS ESP grafana plugin is a data source plugin that allows you to stream data from SAS Event Stream Processing (ESP) servers in a Kubernetes environment to Grafana dashboards With this plugin, you can visualize and monitor real-time streaming data from ESP projects and windows, and apply filters and aggregations to customize your data views. The plugin supports both ESP Pub/Sub and REST APIs, and provides a user-friendly interface to discover and select ESP servers, projects, and windows. The plugin also enables you to use Grafana alerting and annotation features on ESP data. First the SAS ESP plugin must be used to create a connection to the SAS ESP Studio which will be running the project. It looks like this:   Grafana ESP Plugin   Simply enter the url of the ESP Studio which is running the project and save and test your connection. When the connection is made create a Dashboard and start adding visualizations. The visualization edit panel will include a section called data source. Pick the data source which was created above and then select the ESP window and fields which are to be used by this visualization. The visualizations used above are: Base64 Image/Video/Audio/PDF Bar Gauge Gauge Pie Chart Here is an example of how you might configure the SAS Event Stream Processing data source for the bar gauge.   Data Source Configuration   Variables from the output schema of the ESP window which you are connecting to are selected. These variables are then used to Grafana to product the graph.   Conclusion   As you can see, SAS Analytics for IoT provides you with all the tools you’ll need to quickly go from concept to production. Although this example consumed image data, you can use any type of data. This comprehensive solution also provides tools to maintain and govern model development, as well as everything you need to visualize and cleanse new data sources. Let’s see where your imagination takes you! I’d love to hear how you put SAS Analytics for IoT to work for your company. Download the files and try it yourself! >> Resources on GitHub <<  Additional Resources Additional resources might include the following: SAS Analytics for IoT SAS Visual Data Mining and Machine Learning SAS Event Stream Processing SAS for Developers SAS Deep Learning Python Interface SAS Scripting Wrapper for Analytics Transfer   ... View more

What if I told you that you could create a  small town complete with potholes in virtual reality and then use it for computer vision model training?   Would you believe it possible?  The benefits of training in VR are many: Ability to rapidly control and change environmental factors like the change of seasons, rainfall, snow even time of day. Vastly lowers the cost of physical assets during ideation and prototyping. Data collection and training now takes seconds instead of days or months. Ability to model when access to the physical environment or situation isn't possible.  Pairing SAS Analytic for IoT with the Epic Games Unreal Engine and Microsoft's project AirSim the seeming impossible becomes possible.  Check out this video to learn more:     Potholes are bad The financial impact of potholes and other effects of under-maintained roadways has a real impact on the wallets of motorists across the U.S. Each year potholes are causing an estimated $3 billion in blown tires, busted axles, and other collateral damages.   Potholes   Municipalities have developed some ways to prevent potholes from forming in city streets like crack sealing, road patching, and overlays, and these preventative efforts can help reduce damage to vehicles and costly roadway repairs. The main problem is that the same municipalities don't have ways to timely identify the areas that need intervention. This demo shows how visual detection technology can dramatically help cities keep road maintenance costs down and improve overall public safety by identifying and ranking potholes based on their condition so that repairs can be pre-emptively executed. Usually, garbage trucks rumble round the city streets doing a simple job – providing and transporting collected trash. The idea is to convert these humble, single purpose trucks into a road maintenance information gathering hub. By fitting cameras at the front/back of garbage trucks and having an SAS Event Stream Processing engine running with a trained model to identify potholes. Information about the pothole like size and location can then be sent in real-time to a central hub or stored in a database and presented to repair crews in an orderly way. More traditional analytics can then be run there to decide which section of roads can be targeted to be patched first to minimize effect on traffic and maximize crew time utilization.   The solution   Components   For this solution we will use a virtual environment to simulate a small town, due to the fact that it is not feasible to mount cameras on real trucks. An open source project called AirSim will be used. AirSim is a simulator for drones, cars and more, built on Unreal Engine . It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. For this demo we built a virtual small town which includes streets filled with potholes.   The AirSim plugin was added to this environment which allows for the capture of images from the cameras which are attached to the vehicles which traverse the town using a python API.  Model training is done by simply replacing the real world components for the collection of images with our virtual stack.  It all looks something like this:   Training Architecture   A python API is used to control a vehicle which traverses the virtual environment. Attached to the vehicle are cameras which are used to collect images of the town's road conditions. These images are then used by SAS VDMML to train an object detection model. Once trained the model is deployed into SAS ESP to score new images as they are created. The simulator data also includes location information as well as the images. Using a combination of computer vision and location information we can create a database which contains current road conditions and plot them on a map using SAS Stream Viewer.   The output from the training exercise becomes an input to the ESP Build Process.  ESP is where the real time images and data are processed to generate the dashboard.  The demo conceptualizes the use of 4 separate garbage trucks. Data from truck number 1 is collected from the virtual town live using the virtual camera mounted to the truck. Trucks 2-4 represent data previously collected and stored in a data file. Each truck is represented on the map as a different color circle. The collected images will be inferenced using a model trained by SAS using DLPY.    Execution Architecture   The left box represents our world.  An ESP edge server which contains the trained model inferences images in real time.  When a pothole is detected the results are formatted along with the truck ID and geolocation data and sent over MQTT to the ESP server running in the cloud.   The cloud server collates data from each virtual truck into the dashboard.    Conclusion   Visual detection technology is an effective way to detect potholes and help municipalities keep road repairs costs down while improving safety conditions. The benefits extend beyond road repairs, as preemptive intervention can also help shield local administrations from costly vehicle repair bills.  Leveraging a virtual environment helps reduce costs  and speeds up the model training process.  It is a great first step in the model building process and can be augmented when possible with real world images when available.  ... View more

Split String by Delimiter You may not have noticed but as of September 2021 you can now use Lua directly from many ESP windows.  There is even a whole new window type called Lua and you can code almost anything you like in there.  In this article we will create  a custom function which can be used to split an incoming string by a user defined delimiter. For fun we will also create a module for various string related functions and show how these modules are included in your programs.   Overview Let's say you have the following string: https://gitlab.sas.com/IOT/tutorials/esp-lua-code-examples/ You would like to parse out the 3rd word of this string where a word is anything delimited by the "/" symbol. Lua does not provide a straight forward way of parsing string in this way. Moreover, Lua does not support regex so we can't use that either. Instead of using regex, the Lua string library has a special set of characters used in syntax matches. Both can be very similar, but Lua pattern matching is more limited and has a different syntax. If you are like me and find regex infinitely confusing, this dumbed down version might just hit the sweet spot. Detailed information can be found on this tutorial website. The Lua character classes look like this:   Character Classes   As with regex you will also need the ability to specify modifiers that help with repetition of patterns. Now let's use these pattern matching classes to create a function that will split our string on a word boundary. Split Function To make things easy I have written the following split by delimiter function. Let's take a look at it and try and decipher just what it is doing. local function splitbydelimiter(    s,   -- string value    delimiter,  -- string value with defines the delimiter. default is ","    item )   -- number: (optional) item in array to return.               -- Let's say you want the 3rd item parsed from string "item1,item2,item3"             -- If you say 2 item2 will be returned.               -- Note that Lua arrays start at 1 and not 0.      -- returns table of split items or item entry    answer = {};    delimiter1 = delimiter or ","    for word in string.gmatch(s,"(.-)"..delimiter1) do        table.insert(answer, word);        print ("word result...............................", word )    end    print ("answer .... ", #answer)    if #answer == 0 then result = nil    -- return nil if nothing found.    elseif item then  -- does the user want an entry verses a parsed table?        if #answer >= item then            result = answer[item]        else result = nil            end    else  result = answer        end return result; end   The function requires only 1 parameter which is the string to parse and returns either a table of parsed words or an individual item. Programmers choice here. If nothing is found it returns nil. I always like my functions to return nil because it makes check for valid results very easy. For example, if splitbydelimiter('IOT/tutorials/esp-lua-code-examples/') then    -- we found something end The optional parameters are the delimiter which defaults to ',' and the item. If an item number isn't set then a table is returned which contains each word in order. The heavy lifting of the function is done with this line. for word in string.gmatch(s,"(.-)"..delimiter1) do We are passing gmatch 2 things, the string to parse and the pattern. Let's break it down ".-" Matches any character and 0 or more repetitions of any character () The sub pattern within the pattern you want to capture. We want to capture the word without the delimiter. '..' In Lua this is string concatenation. Our pattern will be the first string concatenated to the contents of the delimiter1 variable. If we use a delimiter of '/' our pattern passed to gmatch would be "(.-)/" if our string was "https://gitlab.sas.com/IOT/tutorials/esp-lua-code-examples/" it would break down as follows: https: blank gitlab.sas.com IOT tutorials esp-lua-code-examples nil ESP Example Now let's put this code in an ESP project and see what happens.  For testing I simply created 2 windows. A Timer window that generates an event once per minute and a Lua window with a hardcoded string. We call back to our local function which is typed directly into our main code.   ESP Lua Window The output of 2 timer pops looks as follows:   ESP Test   It is hard to tell the difference using ESP studio between a blank and a nil value. So let's add a couple lines of code to our project and see if it becomes more clear. if event.word2 == nil  then event.word2 = "nil"   else event.word2 = "blank"     end if event.word7 == nil  then event.word7 = "nil"  end   ESP Test II Now it is more clear that word2 is set to something other than nil. While the function returned nil because there is not a 7th word in our string. Isn't coding great? Using Modules If you have code which you want to reuse many times it is good to group them into modules and then just import those modules at the top of your Lua program. Much like the import function in Python except Lua calls it require. You can use modules to create classes which define objects or you can simply use them to group like minded functions together in a logical fashion. In this example, I will group a few string type functions into a module called TTStrings and include that at the top of our program. local tttstrings = require ("/export/sas-viya/lua/TTstrings")  -- group of string functions local TTstr = tttstrings:new() ​ function create(data,context)    local event = {}        local teststring = "https://gitlab.sas.com/IOT/tutorials/esp-lua-code-examples/"    event.id = data.id    event.word1 = TTstr:splitbydelimiter(teststring,"/",1)    event.word2 = TTstr:splitbydelimiter(teststring,"/",2)    event.word3 = TTstr:splitbydelimiter(teststring,"/",3)    event.word4 = TTstr:splitbydelimiter(teststring,"/",4)    event.word4 = TTstr:splitbydelimiter(teststring,"/",4)    event.word5 = TTstr:splitbydelimiter(teststring,"/",5)    event.word6 = TTstr:splitbydelimiter(teststring,"/",6)    event.word7 = TTstr:splitbydelimiter(teststring,"/",7)        if event.word2 == nil  then event.word2 = "nil"      else event.word2 = "blank"        end    if event.word7 == nil  then event.word7 = "nil"  end        return(event) end This line pulls the TTstrings.lua module into memory and gives you a reference to it. local tttstrings = require ("/export/sas-viya/lua/TTstrings")  -- group of string functions This line creates an instance of the class and assigns that to another variable. All the functions inside that module are now available using the TTstr variable. local TTstr = tttstrings:new() Summary Lua is a very powerful addition to the ESP tool bag.  It is fast and provides many facilities for table manipulations, math operations, xml and json strings management.  Best of all as you have seen here it is very easily extendable to whatever your needs require.   ... View more

When building critical applications which require continuous operation, these systems must eliminate single points of failure. This is achieved by creating redundancy at all levels. For example, hospitals and airports require high availability to perform daily activities. When a portion of the IT infrastructure is broken the end users must still have access to key systems to do their work. The idea is to ensure uninterrupted processing of data in the event of a failure within a business critical environment.   There are three message buses supported for ESP failover: Solace, RabbitMQ and Kafka. This video will show how to use SAS Event Stream Processing (ESP) in conjunction with a Kafka message bus to build a real time fault-tolerant application which provides analytic services on a Kubernetes system running Viya 4.   The presentation highlights the following topics: Define high availability Learn the basics of the Kafka message bus Learn how to deploy an ESP project in failover mode Show an end to end example SAS ESP and Kafka work together to create a great high availability system with redundancy built into each part of the system.  When combining these two technologies, real-time data pipelines and advanced analytics come together into a fault-tolerant, fast and scalable system.   Be sure to view my video for more information! ... View more

Companies who are using Microsoft Azure's IoT platform to implement their IoT strategy will likely require the advanced analytic capabilities provided by SAS.  Connecting IoT devices to the cloud is an important first step in any IoT strategy.  However, it is important to leverage the data collected to drive business transformations.  The data collected must be explored, analyzed and modelled.  The resulting models must be then deployed so that machine learning can be applied to the live streaming data.  For example, let's say you would like to predict flooding events for streams in your community.  Weather information is a key ingredient into the prediction of flooding events.  Therefore, in this article we are going to talk about how to query weather data from the Azure Maps Weather Service via a logic application.  The logic application will also post this information to an Azure Event hub so that is can be consumed by SAS Event Stream Processing via a Kafka connector.   Overview In this article  we will configure an Azure Logic application to query the Azure Maps Weather service once an hour.  The logic app will then format the data returned from the weather API into the required JSON string and send it to an Azure Event Hub. Once in the Event Hub, it is available to any application running a Kafka listener.  An ESP project will be created which will retrieve these events from the Event Hub using an ESP Kafka connector.  Once in ESP a function window will be used to parse the incoming JSON into an ESP schema.   Live Weather Data   For this example we will work with the Azure Maps Weather service which will return a JSON message in response to an API call. Please follow the directions described on the Azure Maps web page to request a subscription key which will be used to request weather data. An example API call might be as follows: https://atlas.microsoft.com/weather/currentConditions/json?api-version=1.0&query=47.60357,-122.32945&subscription-key={Azure-Maps-Primary-Subscription-key}    Which would generate a JSON response such as this:   { "results": [ { "dateTime": "2020-10-19T20:39:00+00:00", "phrase": "Cloudy", "iconCode": 7, "hasPrecipitation": false, "isDayTime": true, "temperature": { "value": 12.4, "unit": "C", "unitType": 17 }, "realFeelTemperature": { "value": 13.7, "unit": "C", "unitType": 17 }, "realFeelTemperatureShade": { "value": 13.7, "unit": "C", "unitType": 17 }, "relativeHumidity": 87, "dewPoint": { "value": 10.3, "unit": "C", "unitType": 17 ...   Logic Application    Consider the following logic application flow:    Logic App Flow At a high level there are three parts to this flow. First is a timer which triggers the flow once an hour.  Next variables are initialized.  Lastly, we loop through a list of locations and issue a weather API call for each.  Since Azure is a code first environment you can easily recreate this logic application in your Azure tenant by cloning the provided example from GitHub (entire code block not listed inline for brevity).   { "definition": { "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#", "actions": { "Initialize_array_of_locations": { "inputs": { "variables": [ { "name": "weather_locations", "type": "array", "value": [ "35.79, -78.78", "35.78, -78.75" ] } ] }, .................   Once the logic app example is cloned into your tenant you will need to customize it to contain your location and API key.   Locations   Edit the location array and change these values to your latitude and longitude. { "name": "weather_locations", "type": "array", "value": [ "35.79, -78.78", "35.78, -78.75" ] }   Weather API key   Also edit the following key to match your subscription:  "subscription-key": "ssRYdF8oylxr-B5_N0a7SUwOHlbGk6TsesTnV53Y"   Event Hub   Next we need to configure an Event Hub to process events. Azure uses the term Event Hub to mean both the cluster and the topic and it gets confusing so let's go over the conceptual mapping of terms between Kafka and Event Hub. Event Hub to Kafka mapping When you create a Kafka environment it is created by defining broker and zookeeper nodes to create a cluster.  Azure does not expose the zookeeper concept. Therefore, creating our Kafka cluster will simply consist of creating a namespace and an event hub.  This corresponds to creating a cluster and a topic in Kafka.  First create the namespace.   Create Namespace Next create the event hub or topic.   Create topic Max message retention is 7 days. This means you will be able to replay 7 days' worth of messages from the event hub if needed.  Increasing the partition count will increase message throughput. Since we are only producing 1 message per hour I will leave this at 1.   Connecting the Logic App to the Event Hub   Now that we have 2 Azure resources we need to connect them together so that messages flow between them.  Navigate back to the logic application created earlier and edit the Send event action.  Create a new connection using the namespace and event hub that was just created.   Send Event  Now your logic app is actively issuing calls to the weather API, parsing the responses into a JSON string and sending that information to the Event Hub.  Next we configure the ESP project to retrieve the messages from the Event Hub.   ESP Project   Our ESP project will consist of 2 windows.  First will be a source window which will contain our Kafka connector.  Source windows provide a means for ESP to ingest data.  Next will be a Function window which will parse the JSON string into the schema.   ESP Project Kafka Connector   In this example I have set up 2 Kafka connectors. The first will keep track of message offsets and retrieve only the latest message which has not been previously read.  The second will read all 7 days' worth of messages that are stored in the Kafka topic.  The difference is between needing current or historical data.   Connectors   Let's take a closer look at the project XML to see how a connector is configured.   <connector class="kafka" name="kafkain7days" active="false"> <description><![CDATA[Grabs 7 days' worth of data]]></description> <properties> <property name="type"><![CDATA[pub]]></property> <property name="kafkatype"><![CDATA[opaquestring]]></property> <property name="kafkainitialoffset"><![CDATA[smallest]]></property> <property name="urlhostport"><![CDATA[notusedwhennotHA]]></property> <property name="kafkahostport"><![CDATA[Namesspace-weather-catcher.servicebus.usgovcloudapi.net:9093]]></property> <property name="kafkatopic"><![CDATA[eh_weather_catcher]]></property> <property name="kafkapartition"><![CDATA[-1]]></property> <property name="kafkaglobalconfig"><![CDATA[security.protocol=SASL_SSL;sasl.username=$ConnectionString;sasl.password=Endpoint=sb://namesspace-weather-catcher.servicebus.usgovcloudapi.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=J5qKcs1mdF7vTbJZ7FNsNZBxlfsAHa3uLzuqI3Yo=;sasl.mechanism=PLAIN;ssl.ca.location=/etc/pki/tls/cert.pem]]></property> </properties> </connector>   The namespace and event hub defined previously are mapped to their Kafka counterparts in the connector. type pub indicates messages are published to this ESP opaquestring indicates all message content is contained in one string variable kafkainitialoffset set to smallest will get every message available from this broker urlhostport is normally set to the zookeeper node which is unavailable in Azure kafkahostport refers to the Kafka broker URL topic and event hub are synonymous kafkapartition = -1 means retrieve messages from any available partition kafkaglobalconfig defines the security and access used to talk to the broker You must retrieve the sasl.password from the namespace defined in Azure. First navigate to the namespace defined earlier and then click Shared access policies. Copy the connection string-primary key into the ESP project. For example:    JSON Decoding via ESP Function Window   Since we parsed the weather data responses in the Azure Logic Application the JSON messages returned from the Event Hub contain only the information we require.  All we need to do now is parse the variables into the ESP schema.  The function windows is defined as follows:   <window-functional pubsub="true" index="pi_EMPTY" name="func_parse_json"> <description><![CDATA[Parse incoming json into fields.]]></description> <schema> <fields> <field name="index_esp" type="int64" key="true"/> <field name="deviceID" type="string"/> <field name="timestamp" type="stamp"/> <field name="Latitude" type="string"/> <field name="Longitude" type="string"/> <field name="humidity" type="int32"/> <field name="rain" type="double"/> <field name="rainplus1" type="double"/> <field name="temperature" type="double"/> <field name="windspeed" type="double"/> </fields> </schema> <function-context> <properties> <property-json name="jsonData"><![CDATA[$message]]></property-json> </properties> <functions> <function name="deviceID"><![CDATA[json(#jsonData,'deviceID')]]></function> <function name="timestamp"><![CDATA[json(#jsonData,'timestamp')]]></function> <function name="Latitude"><![CDATA[json(#jsonData,'Latitude')]]></function> <function name="Longitude"><![CDATA[json(#jsonData,'Longitude')]]></function> <function name="humidity"><![CDATA[json(#jsonData,'humidity')]]></function> <function name="location"><![CDATA[json(#jsonData,'location')]]></function> <function name="rain"><![CDATA[product(json(#jsonData,'rain'),.0393700787401575)]]></function> <function name="rainplus1"><![CDATA[product(json(#jsonData,'rainplus1'),.0393700787401575)]]></function> <function name="temperature"><![CDATA[json(#jsonData,'temperature')]]></function> <function name="windspeed"><![CDATA[json(#jsonData,'windspeed')]]></function> </functions> </function-context> </window-functional>   Writing Messages to an Event Hub   Now let's say we would like to return messages to an output topic called eh_weather_slinger from ESP.  To do that simply create another event hub or topic called eh_weather_slinger in the namespace called namesspace-weather-catcher as described above.  ESP connectors which publish data outbound are denoted with the type sub which is short for subscription.  We will also change the kafkatopic property to identify our newly created outbound topic or event hub. Our connector description will be as follows:    <connector class="kafka" name="kafkaoutbound" active="false"> <description><![CDATA[Publish data to outbound topic ]]></description> <properties> <property name="type"><![CDATA[sub]]></property> <property name="kafkatype"><![CDATA[opaquestring]]></property> <property name="kafkahostport"><![CDATA[Namesspace-weather-catcher.servicebus.usgovcloudapi.net:9093]]></property> <property name="kafkatopic"><![CDATA[eh_weather_slinger]]></property> <property name="kafkaglobalconfig"><![CDATA[security.protocol=SASL_SSL;sasl.username=$ConnectionString;sasl.password=Endpoint=sb://namesspace-weather-catcher.servicebus.usgovcloudapi.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=J5qKcs1mdF2rZ7vTbJZ7FNsNZBxlfsAHa3uLzuqI3Yo=;sasl.mechanism=PLAIN;ssl.ca.location=/etc/pki/tls/cert.pem]]></property> </properties> </connector>   What have we learned?     How to drive messages into an Azure Event hub from a logic application How to use the ESP Kafka connector to connect to an Azure event hub How to use a function window to transpose a JSON message into ESP variables How to write data back to the event hub or topic    This is the first step in gathering weather forecast data for your streaming machine learning project.  Next add some sensor data and some trained machine learning models and start predicting flooding before it happens.   ... View more

Imagine you are responsible for manufacturing quality at your company. Wouldn't it be great to know what percentage of product contains flaws? Maybe replace human oversight with automation or collect data to perform a root cause analysis. SAS Analytics for IoT is a perfect solution to solve this or any Computer Vision (CV) problem. This comprehensive AI-embedded solution provides so many capabilities because it integrates a number of key SAS products (see the figure below).   SAS Analytics for IoT Here we are going to simulate our manufacturing process using a Rubik's cube. Our cube represents a part which is being produced, and it is up to us to determine whether it was manufactured correctly.  This will be accomplished using an image classification computer vision model.  This article will cover the following topics:  Gathering and classifying training data Train CV models using the SAS Deep Learning Python (DLPY) package Operationalize our models using streaming analytics and ESPPy Please refer to the github repository iot-image-classification-rubiks-cubes for more information and examples.    Before we get started please take a look at look at the following video which gives an overview of the process:    First off, let's talk Computer Vision CV techniques provide the ability to acquire, process and analyze incoming images. This analysis produces numerical results in the form of predictions based on the classes we define. In this example, we need to create one CV model which will be used to tell if a Rubik's cube is solved or unsolved. We will also add in a third class which will represent any image where the cube isn't properly presented to the camera. We don't want to score images where there is no cube present or the camera can only see one side of the cube. Having an invalid case makes the predictions more accurate. Since we are using image classification without object detection our cubes must be correctly positioned within the camera frame in order to get accurate results. Here is an example of a correctly positioned cube: Looking Good Ideally, three sides of the cube should be visible to the camera and you would need two cameras to know for sure that the cube is correctly manufactured. However, for this example I am considering two visible sides valid, and simply holding the cube in my hand as I wait for the image to be scored.   Creating a training dataset The first step in the process is creating an awesome training dataset. When building your dataset remember these axioms: A Computer Vision model is only as good as the dataset it was trained on If you can't see it in the picture, the computer won't be able to tell either If possible collect images from the production environment It is recommended that no less than 1000 images be used for each class you intend to train and more images are always better. These images are also subdivided into two datasets. The first is for training and the second is a testing dataset. Therefore, an additional 200 images for testing is also recommended. Image size should be no less than 224 by 224 pixels.  You may need to add your own images to the training dataset. For example, I was told that Rubik's cubes in Brazil have different colors.  For my training, I collected 1000 images of the 3 classes, good, bad, and empty.  Each class of images needs to be separated into its own sub directory.     Next, we train our models Before we build an application that uses analytical models to predict outcomes, we need to train them. The training process for CV involves classifying images and separating these images into datasets that can then be fed into a machine learning model such as ResNet50, VGG16, or Darknet. This stage of the process is completed using the SAS Deep Learning Python (DLPy) package which provides the high-level Python APIs to the deep learning methods in SAS Visual Data Mining and Machine Learning (VDMML). Training Flow   The previous diagram illustrates the flow for training a deep learning analytical model. Starting from the left we see that a Jupyter notebook is used to issue DLPy and SWAT commands which communicate with SAS Viya. For large jobs Viya has the ability to farm processing out to workers and take advantage of parallel processing. GPU support is also available. Deep learning algorithms are then invoked as each image is processed to create a portable analytics file called an .astore file. A typical training exercise contains these steps: Setup libraries and launch CAS Load and explore the training data Prepare the data for modeling Specify the model architecture, configure model parameters and import pre-trained weights Fit the image detection and classification model Evaluate the newly created image classification model Visualize model results Save model as .astore for deployment For more information about the DLPy high-level Python APIs which allows you to build deep learning models please see the following examples: Image Classification Object Detection Now use streaming analytics Streaming analytics is defined as the ability to constantly calculate statistical analytics on an incoming stream of data. In our case, that stream of data is the images coming from the camera. SAS Event Stream Processing (ESP), which is part of the SAS Analytics for IoT solution, provides the ability to deploy our newly trained analytical model, in the form of an .astore file, at the edge. With ESP you can ingest, filter and transform your data in stream. Streaming Flow   Here we can see the flow of information through the system and highlight some key points: ESPPy adds flexibility In this diagram we are showing the flexibility of using ESPPy to remotely connect to ESP from a camera which is not directly connected to the ESP server. Here we are using OpenCV to collect and transform the image before it is sent to ESP for processing. ESP is cloud enabled Depending on your production environment, ESP can be placed at the edge or in the cloud. ESP Includes a powerful graphical development platform. Using ESP Studio models such as these may be created without any coding. ESP operationalizes analytics Expertly trained models are now available anywhere, assembly line, factory floor, anywhere CV is needed. As seen here, the ESP model which creates our detector is quite simple.  All that is needed is a source window to inject images, 2 windows to load the CV model and a scoring window.   Once the model is loaded and running a Jupyter notebook can be used to visualize the outputs from the imageScoring ESP window.   ESP Project Using ESPPy with Jupyter notebook Using a Jupyter notebook we can use ESPPy to display ESP project information and subscribe to the imageScoring window to see the results. Let's take a look at the ESPPyRubiksControl Jupyter notebook which is located in the streaming directory. Using this notebook you can, Connect to ESP Server Load Project Subscribe to Event Streams Visualize Data from Streams After the notebook is running in Jupyter connecting to an ESP server is a simple one line command. Once you have a connection to the server, you can query it for information about projects (running and stopped) and the server itself. You can also create, load, start, stop, and delete projects. Projects can be loaded from file paths, Python file objects, or URLs. Using the project.to_graph(schema=True) we can graphically see the currently active project. Project to Graph Jupyter notebook and ESPPy can be used to show streaming statistics. For example, here is a bar which shows the recent Rubik's cube results: Streaming Results The combination of ESP, ESPPy, Python and Jupyter notebook is quite powerful.  As you can see, SAS Analytics for IoT provides you with all the tools you’ll need to quickly go from concept to production. Although this example consumed image data, you can use any type of data. This comprehensive solution also provides tools to maintain and govern model development, as well as everything you need to visualize and cleanse new data sources. Let’s see where your imagination takes you! I’d love to hear how you put SAS Analytics for IoT to work for your company. ... View more

Have you ever wondered how to collect streaming data from your phone, pass it to SAS's streaming analytics engine and use the power of unsupervised machine learning models to construct an anomaly detector?  If so, you are in luck. This article will show how to collect live accelerometer and gyroscope data from a mobile device and pass it to Event Stream Processing (ESP) using an MQTT protocol. Once in ESP, the JSON data is decoded and passed to the unsupervised machine learning models k-means and subspace tracking (SST). These machine learning models find patterns in data and are trained dynamically from streaming data.     This article will go into quite a bit of detail on how this is all accomplished.  However, if you are like me and can't be bothered with the details, you can just watch the following video:     OK, you have watched the video and are still reading, great!   That means you are a detail oriented person. Here is what we are going to cover: Use Case Architecture Mobile Application Installation  Configuration ESP Project details  Function Copy Aggregate K-means Training K-means Scoring SST calculate Join Anomaly Detection Example Summary More information is also available on this github page.    Use Case In this use case, we will consider the mobile device a sensor that can be placed on any stationary piece of equipment on a factory floor. For demo purposes our anomaly detection time frame will be 30 seconds. In a real world scenario, this time frame will be much longer. That is to say as long as the equipment continues to generate 30 seconds of consistent readings, this will be considered the baseline. Once that 30 second baseline is established, new observations will be compared against it and any deviation will be considered an anomaly. KPIs will determine the severity of the anomaly. This type of self-training is awesome because the baseline is dynamically set and does not have to be calculated separately for each piece of equipment. If the sensor is placed on a rotating object, the accelerometer data being produced becomes the norm as long as it is consistent. Imagine a factory floor with 1,000 machines each oscillating at different amounts. Now imagine how much time would be saved if the sensors placed on these machines were able to self-train their own unique anomaly detection models. Very powerful.   Architecture   Because MQTT is one of the more popular IoT protocols we are going to use it along with the message format JSON to communicate both bound and outbound between the mobile device and ESP. Consider the following diagram:   You may use your own MQTT broker or a freely available one. As long as ESP and the mobile device can reach the broker communication is established. This architecture allows for the data source to be abstracted to where the mobile device data can come in a variety of ways without needing to modify the ESP model.   Mobile Application   The mobile application collects data from various phone sensors such as accelerometer, gyroscope and location. The sensor data is by default collected at 10 events per second and formatted into a JSON string and then sent to the MQTT broker defined and active. The Android application files may be downloaded from this github location.   Configuration   Configuration is a two-step process. First, configure your broker settings to match your ESP project. Second, select Anomaly Detection from the navigation panel. Here is an example of the broker list. You can see from the green bar that the application is actively connected to the sample MQTT broker. You may define as many brokers as you need. However, remember that the broker name must match the ESP project MQTT connector definition.     Here is an example of the sample broker definition. The broker test.mosquitto.org should always be active.The mobile application will publish to the topic PhoneSensors and subscribe to the topic Outcomes. These topic names may be changed. However, the corresponding ESP project will also need to be updated to match the new names.     Once you have confirmed that the broker is correctly configured and active select Anomaly Detection from the navigation panel to view the results. If the ESP project is running, your "Anomaly Detection" panel will start updating in 30 seconds.   ESP Project   Now let's take a look at the ESP project that will consume the phone sensor data.     Source    Data flows from the top of the diagram starting with the blue source window. The small icon in the lower left of the source window indicates that this source window contains an input connector. If we take a closer look at the connector configuration we can see the host, and inbound topic name. These items must match the settings in the mobile application broker settings. The mqttmsgtype indicates all the JSON data will be placed in one opaqestring. The next step will parse the JSON data in the string and place it into various ESP variables.     Function   The next window is called a function window and will parse the input message into ESP variables. The first step happens in the functions properties section. This is where the opaqestring called $message is loaded into a field called jsonData. The JSON data is then parsed in the functions section into variables. These variables are part of the output schema for the next step in the process. Here is an example of the parsing syntax: In this example, the Name column contains a value that matches the output schema for this function window. This is the parsed JSON value that will be passed to the next downstream window. The Code column contains any function you would like to run. A complete list of available functions can be found in the Event Stream Manager documentation. Here we are using the "json" function to parse the variable "#jsonData" for various strings. You may note that the JSON string may contain more data than our flow requires. This parsing method provides a method for ignoring superfluous data. You may also notice that variable names may be remapped to more meaningful names. Here I am remapping the JSON variable y_acc to an ESP variable called yGravity.   Copy   In order to do in-memory calculations on data we must create retention. Retention is loading a set number of records either by time or count into memory so that they can be used for aggregation and dynamic modeling. We do this in a copy window. Here we are specifying 300 records be saved in memory. Since the mobile application is sending in 10 records per second. 300 records equals 30 seconds worth of data.  Note that the lightning bolt in the lower right of the window indicates that this window maintains state or retention.   Aggregate   The next window is an Aggregate window. Since we now have 300 records in memory we can start to use that data to create summary statistics. Using the aggregation window we will create 6 new variables. We will create a standard deviation and average for the X,Y and Z Gravity data sent in by the phone's accelerometer. These values are updated each time a new message is received on our MQTT topic. Only the newest 300 records are used in our calculations.   K-means Training   The train window receives all the phone sensor data and uses it to dynamically create a model based on the analytical model specified, in this case, K-means. Other types of models are available such as: Linear Regression SVM (Support Vector Machine) Logistic Regression Recommender TSNE (T-distributed Stochastic Neighbor Embedding) K-means is a clustering method which aims to partition observations into groups. For example, imagine shooting 100 arrows into a target. Your arrows will form clusters or groups on the target. Each group will have a geographical center point. If you shoot one more arrow, represented here as the green dot, you can then measure the distance from the center point to the location of the new arrow. This distance is a measure of the deviation from the baseline or norm. In the diagram the red circle represents the mathematically defined cluster based on the dispersion of data in the scatter plot. Here the red arrow represents the distance from the center of the circle to the new data. This of course is a simplified view. Keep in mind that our accelerometer data is 3 dimensional and contains data points for the X, Y and Z values.     The training window will use this algorithm to create a model and then pass that model to the scoring window to calculate the distances from the norm.   K-means Scoring   The dynamically trained model is passed to the scoring window from the training window and used to score each new event. In our phone sensor example, we might place the phone on a flat surface and leave it there for 30 seconds. Since the phone is not moving all the accelerometer readings will be nearly zero. Zero becomes the center point of our k-means cluster. As long as the phone does not move the distance from the norm is also zero. If I start shaking the phone the new events will deviate from the baseline greatly and the distance value will increase. K-means only works as an anomaly detector because our use case produces very consistent values. Next let's look at the more robust algorithm subspace tracking.   SST Calculate   Calculate windows are designed for data normalization and transformation methods, as well as for learning models that bundle training and scoring together. Here we are going to use a subspace tracking algorithm also known as moving window PCA (principal component analysis) to determine deviations from the norm. The k-means algorithm attempts to group data points into clusters and measure the distance from the center of the cluster. Principal component analysis reduces the dimensionality of the data from n variables to m principal components. Since we are bringing down the dimension of the original data we are creating a subspace from the higher space and this is how the name is derived.   In SAS ESP, there are 5 metrics available to track the subspace: PCAngleChange, PCAbsoluteAngle, projection angle, rank and residual.  For this example we are going to concentrate on the PCAngleChange metric, which is based on the first principal component. When the data creates principal components which fall into the same line they are mapped along a vector. Vectors have a direction and are plotted along an axis.  When a new observation arrives, the new direction is recalculated and the angle between the previous direction and the new direction determines the deviation. Therefore, we determine the deviation by measuring the amount of angle between the expected vector and the current one. The larger this angle, the more the possibility of the new observation to be anomalous. Subspace tracking will continually calculate the expected PC vector using our moving window of 300 events and provide a basis for comparison.     If we look at the scatter plot data using SST, instead of a cluster we derive a vector shown here in red from the principal components of the accelerometer data. When a new data point, shown here as a green dot is received, a new PC vector is created. Shown here as the blue vector. The angle between the red and blue vectors is the amount of deviation from the baseline. This information will be passed to our mobile application where thresholds are checked and will identify if the new observation is an anomaly. Our vectors are calculated dynamically from a moving window of 300 samples. As with k-means, this is a simplified view of what is actually happening. Vectors are by nature 3 dimensional and eigenvectors must be used to tune the model. Here are values used to train this model. Here we can see our window length, input fields and eigen settings.     The output map references principal components, angles and our 3 vectors.     Join   Now that all the data science is done we need to join all the information into one event and then send it back to the MQTT broker. The broker will then deliver it back to our mobile device to be displayed. From the following picture you can see our newly combined data includes fields calculated by SST, k-means and summary statistics.     On bottom right of the join window you will see a connector icon. This indicates that this window contains a connector. In this case an MQTT connector. This connector will JSON encode the data from our output schema and send it to the MQTT broker.   Anomaly Detection Example   Now let's take a look at the use case where the mobile device is a sensor and has been placed on a stationary piece of equipment on a factory floor. For demo purposes our anomaly detection time frame will be 30 seconds. At this point we have our ESP project running and our mobile device is connected to an MQTT broker and is receiving accelerometer data. The ESP project analyzes and improves the data and then sends results back to the MQTT broker topic called "Outcomes." The mobile application includes thresholds which determine warning and error levels. We start by selecting anomaly detection from the navigation panel and setting the phone down for 30 seconds to establish a baseline for both k-means and SST. Once this baseline is set the anomaly detection will report green. Later we start moving the phone. The anomaly detection panel will turn yellow and then red based on how much the phone is moved.     Summary   This article has shown some key features found in SAS Event Stream Processing. Using ESP you can easily collect data from remote sensors using industry standard protocols. After parsing, we then retained events in memory so that advanced analytical methods can be used to create an anomaly detector. Dynamic model training allows models to be continuously trained based on real time data and is shown here using both the calculate and train windows. Data can be a easy combined using the join window and then published to an outbound topic. The mobile platform provides a level of personal interaction with the data that cannot be easily replicated using canned data sources. It helped me better understand how SST and k-means work in an interactive environment and I hope it will help you as well.       ... View more

Ok, today you’re a SAS user.  But, imagine that you were also the owner of a retail store!  Wouldn't it be great to know what percentage of customers in your store looked happy as they browsed your shelves, or perhaps which underage customers are trying to buy cigarettes?  Applications of machine learning, computer vision (CV) and streaming analytics abound, covering use cases from enhancing real time in-store customer experience, to grid surveillance and security, predictive maintenance, claims fraud management and improved manufacturing quality.    I recently met with an executive from a security camera company and our discussion centered around how easy it is to stream video data into the SAS platform.  This got me thinking, once the data is in the SAS platform, what can we do with it?  After some thought, I decided that gathering and displaying demographic data would be a perfect use case, and that SAS Analytics for IoT was the perfect solution to solve this or any computer vision problem.  This comprehensive AI-embedded solution provides so many capabilities because it integrates a number of key SAS products (see the figure below).     Gathering and displaying demographic data: a retail example Using two products that are embedded in the SAS Analytics for IoT solution, SAS Visual Data Mining and Machine Learning (VDMML) and SAS Event Stream Processing (ESP), I’m going to show you how to train and deploy various demographic analytic models to create a single solution that connects directly to a USB-connected security camera.  With this connection, I can process incoming images using pre-trained analytical models and produce demographic statistics.  This application provides: Face Detection - Process incoming images and detect human faces in the video stream Age Classification - Determine approximate age grouped by category Gender Classification - Determine male or female Emotion Classification - Group by these classes: Happy, Neutral, Sad, Fear, Surprise, Angry   An example sample user interface might look like this:       On the left, I’m showing the demographics as it relates to the current face detected in the security camera video feed in real time.  On the right, I have an example of the total gender statics for that day.  Now let’s talk about the steps you’ll need to build this type of solution yourself.   First off, let's talk computer vision Computer vision (CV) techniques provide the ability to acquire, process and analyze incoming images.  This analysis produces numerical results in the form of predictions based on the classes we define.  In this example, we need to create four separate CV models.  First, we need an object detection model which will not only give us the probability there is a human face in the incoming image, but it will also give us the coordinates of that face in the image, in the form of a bounding box.  Using the box coordinates we can then crop just the face from the incoming image and send that data to the next analytical model.  Consider this example:     Here I’m using a YOLO (You Only Look Once) object detection model to find all the human faces in the incoming video stream.    Next, we train our models Before I can build an application that uses analytical models to predict outcomes, we need to train them.  The training process for CV involves classifying images and separating these images into datasets that can then be fed into a machine learning model such as ResNet50, VGG16, Darknet etc.  This stage of the process is completed using the SAS Deep Learning Python (DLPy) package which provides the high-level Python APIs to the deep learning methods in SAS Visual Data Mining and Machine Learning (VDMML).         As the previous diagram illustrates, four separate datasets were created to support model training.  In cases where a dataset supports many types of image classes, Multitask Learning may be applied.  Here, I tweaked each dataset to get the best possible output which led to keep the training separated.  Therefore, each image dataset supports the training of one model.  Face detection was trained using a Yolo V2 architecture while age, gender and emotions were trained using a ResNet50 architecture.  For example, when training the gender model, image data is loaded into SAS Viya and VDMML using DLPy.  Deep learning algorithms are then invoked as each image is processed to create a portable analytics file called an ASTORE file.  VDMML is GPU-enabled so that training times are greatly improved.  A typical training exercise contains these steps:   Setup libraries and launch CAS Load and explore the training data Prepare the data for modeling Specify the model architecture, configure model parameters and import pre-trained weights Fit the image detection and classification model Evaluate the newly created image classification model Visualize model results Save model as ASTORE for deployment   Please see the "Sources and more information" section for an example of the gender training model.   Now use streaming analytics Streaming analytics is defined as the ability to constantly calculate statistical analytics on an incoming stream of data.  In our case, that stream of data is the images coming from the camera.  SAS Event Stream Processing (ESP), which is part of the SAS Analytics for IoT solution, provides the ability to deploy our newly trained analytical models, in the form of ASTORE files, at the edge.  With ESP you can ingest, filter and transform your data in stream.  I like to think of it as enabling business rules to gain extra business value.  For example, let's say your company wanted to track how many happy women between the ages of 10 and 30 walked by the frozen food section on Tuesday when you were running a sale, verses a normal day when the sale is not running.   ESP gives you the capability to make that happen.  There is a saying here at SAS, "Without deployment, analytics is only a science experiment."    This diagram illustrates an overview of this project deployment architecture.     Here we can see the flow of information through the system and highlight some key points: ESP is built for speed. Although, there are many methods of ingesting data into ESP (REST, MQTT, MQ), to make this superfast I used a UVC connector which allows me to directly connect ESP to the incoming video stream from the camera.  I also took advantage of ESP's multithreaded capability by scoring age, gender and emotion simultaneously each in its own thread.  ESP integrates with open source. You can easily call python scripts in stream from the ESP model flow.  This allows further integration with other open source packages such as OpenCV.  Using Python and OpenCV, the images were cropped, resized, reshaped and colors were manipulated.  Anything is possible.  Retention is amazing. Retention defines a group of events over a time or event count.  Instead of doing analytics on each image individually, you can now take a group of images and create new event data – like the number of men over the last hour or total number of kids today.  It is very powerful. ESP includes a powerful graphical development platform. Using ESP Studio models such as these may be created without any coding.  For example, publishing my results to MQTT is as easy as dragging a window onto the canvas and filling out an MQTT specifications form.    Let's see analytics in action Now that we have the background, let's take a look at this application in action. In this demo, you'll see how streaming image data is transformed or enhanced in real-time to deliver demographic data.     As you can see, SAS Analytics for IoT provides you with all the tools you’ll need to quickly go from concept to production.  Although this example consumed image data, you can use any type of data.  This comprehensive solution also provides tools to maintain and govern model development, as well as everything you need to visualize and cleanse new data sources.   Let’s see where your imagination takes you!  I’d love to hear how you put SAS Analytics for IoT to work for your company.   Download the files and try it yourself!   >> Resources on GitHub <<    Sources and more information SAS Analytics for IoT SAS Event Stream Processing SAS for Developers Jupyter Notebook reference for gender training SAS Deep Learning Python Interface SAS Scripting Wrapper for Analytics Transfer   ... View more