Deployment of SAS Event Stream Processing Projects using Project Packages
The purpose of this post is to learn how to create SAS Event Stream Processing Project Packages and how to use them to streamline deployment and management of SAS Event Stream Processing projects. Project packages allow us to bundle all the SAS Event Stream Processing project components in a single ZIP archive, making the project portable and easy to copy to different environments. They also make it easier to maintain and update the project components, with built-in Git integration in SAS Event Stream Processing Studio. These project packages are also a necessary step in preparing a project for deployment using the upcoming Project Container feature that allows us to deploy a self-contained SAS Event Stream Processing container with only the components necessary to run a specific project package.
Traditionally SAS Event Stream Processing projects have been XML files representing the flow of data and information through Event Stream Processing windows. Updating the project would require updating the project XML and potentially altering the configuration of files or resources in the deployment environment. This can be inconvenient because the development and testing environment may differ from the production environment, and any changes to the files or resources used by the project will require configuration changes in the development, testing, and production environments. Version control and management becomes challenging, we can manage updates and versioning for the XML file, but the configuration and resources necessary for deployment must be maintained separately from the project XML.
The Project Package feature in SAS Event Stream Processing improves on this traditional approach by including the XML and all supporting files for the project in a single ZIP archive. For the simplest SAS Event Stream Processing projects this could mean just including some test files with the project XML, but for more complicated projects this could include supporting files like model Analytic Store (ASTORE) and ONNX files. These files are required for deployment of the project and must be maintained and updated for reliable performance, but they are stored separately from the project XML. Using the Project Package feature, we can not only store these files in a consistent location with the project XML, but we can perform version control on the entire Project Package, updating individual files in a Git repository to update the whole Project Package.
When creating a new project in SAS Event Stream Processing Studio we have the option to create a project package or just create the project as an XML. Although it is a best practice to create our projects as project packages, it’s straightforward and easy to create a project package later as well. Let’s look at an example of converting an XML project in SAS Event Stream Processing Studio into a Project Package.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Notice in this project that the Project type is an XML file, and in the red box on the left we can see that This project is not packaged. To create a project package, we just need to click the Create package button there in the Project Package pane on the left. The three windows circled with red boxes all have some kind of input file associated with them. When the project is just an XML these files are stored in a Persistent Volume associated with the SAS Event Stream Processing Studio container in the Kubernetes cluster (in this example they live in the /mnt/data directory in the container, but it maps to an underlying hard drive location used by the Kubernetes cluster). Essentially these files must be stored and configured in the deployment environment for everything to work. The geofenceDefinition window and the readVideoData window both use input connectors to read data files from the operating system. The readASTORE window points to an ASTORE file saved on the operating system:
The project won’t execute correctly if the required ASTORE file is not available in the /mnt/data folder in the SAS Event Stream Processing studio container. In this example we need to keep track of these files independently from the project XML and make sure we copy them correctly into the production environment.
We can click the Create package button we viewed earlier to convert the project from an XML file into a project package. After we do this, it creates a directory structure inside the project package where we can store and manage files associated with the project:
The folder structure shown here is created by default when the project package is created. Notice that it includes a blank readme file and a folder to store any project outputs. Although our project had some required files they do not automatically appear here in this package, we must upload any required files into the project package. In this workflow we want to upload the following three files:
- The input data used to test the project, in this case a .avi video file. This file will be saved in the test_files directory since it is used to test the project (in deployment we will use streaming video data for the source window).
- The geofence definition in .csv format used to define the region of interest in the geofence window. This file will be saved in the files directory since it is a supporting file used in the project (it’s not really input data even though we read it using a Source Window).
- The ASTORE for the object detection model used in this SAS Event Stream Processing model. In this case we will copy the pretrained model ASTORE file into the analytics directory in the Project package. ASTORE formats are flexible since they are binary files, but we usually use the format .sasast or .astore.
After uploading the files, we can see them in the Project Package pane:
Of course, you can organize your files in the project package however you’d like, but the developers have created a basic folder structure to guide your thinking in how you store the files in the project package. The next step to adapt the project to the package is to modify any paths pointing to the files so that they now point to the files in the Project Package instead of pointing to the files on the operating system (in our example in the /mnt/data folder in the SAS Event Stream Processing Studio container). The root path for the Project Package directory is specified in SAS Event Stream Processing Studio using the @ESP_PROJECT_HOME@@ variable. As an example, the ASTORE file would be in the location: @ESP_PROJECT_HOME@/analytics/drtaco_yolo.astore. The SAS Event Stream Processing Studio graphical interface also allows us to navigate through the project package folder structure when selecting input files:
The project package is by default stored in an internal Git repository on the SAS Event Stream Processing Studio container (on the Persistent Volume), but you can always switch it to rely on a remote Git repository, which often means using something like Github or an enterprise Gitlab to perform version control and maintain the project. The ability to use Git to manage the project is one major benefit of using the project package feature in SAS Event Stream Processing. Another major benefit is the upcoming Project Container feature that allows the project package to be bundled with the software component necessary to run the project into a single simple container that can be deployed anywhere container will run.
Now that we have created the Project Package, we can create versions of the project in SAS Event Stream Processing Studio to move the project from Development to Test, and then to Production. The key idea about this DEV/TEST/PROD distinction is that projects are created in SAS Event Stream Processing Studio as DEV project, and can be promoted from there to TEST, but to move from TEST to PROD we must use SAS Event Stream Manager. This way we maintain a distinction between the environment where we build and experiment with the project and the environment where we deploy the projects. This whole process is designed to be performed seamlessly within the SAS Event Stream Processing graphical interfaces (building the project in SAS Event Stream Processing Studio and deploying the project to a cluster or edge device using SAS Event Stream Manager), but you can also download the Project Package ZIP archive directly from the SAS Event Stream Processing Studio interface using the Export Project button:
The ZIP archive will contain all the project files you uploaded, along with the underlying project XML. The Project Package feature builds upon the project XML by packaging all components necessary for deployment of the project into a single ZIP archive, including the XML that defines the project.
Project Packages are necessary for deployment using the upcoming Project Container feature as well. This feature creates a specialized container with the Project package and all the compiled SAS Event Stream Processing binaries required to run the project (only those compiled binaries required to run the project, excluding components that are not necessary for the project). This container can then be deployed in a standalone way without requiring any special installation or pre-existing SAS Event Stream Processing footprint. This helps streamline the maintenance and deployment of SAS Event Stream Processing project by creating a single resource that can be updated and deployed.
References:
- SAS Event Stream Processing Studio Project Package Documentation
- SAS Event Stream Processing Studio File-Based Versioning Documentation
- SAS Event Stream Manager Project Containers Documentation (Early Adopter Feature)
Find more articles from SAS Global Enablement and Learning here.
