If you have deployed the new SAS Viya (2020.1 or later)  on Kubernetes you have been exposed to the kustomization.yaml file used for customization of the environment.  And depending on which components you have configured you may or may not have needed to consider the order in which customization items are placed in each of the respective sections.  In this article we look at an issue you might encounter if you configure some of these items and don’t visit the Deployment Notes and reread the README.md files before your next update.   Background The Global Enablement and Learning (GEL) team deploys and manages SAS Viya on long-running RACE reservations for a variety of workshops. In order to facilitate re-building these environments dynamically from scratch on a regular basis we have built a fair amount of automation.   This automation spans many facets of a SAS Viya environment, from ensuring the core software components are installed, such as Kubernetes and nginx, to creating NFS file systems, and ultimately deploying SAS Viya.  Once SAS Viya is deployed additional automation performs more configuration, loads content, creates CASLIBs and loads data, as well as a myriad of other tasks to ensure the required pieces are in place.     An issue at Stable 2020.1.3 One customization task that was automated is the creation of a persistent volume for Python and R. This PV was created to make it available to the SAS Micro Analytic Service, CAS, SAS Compute, etc.  Automation was developed to copy and configure the python-transformer.yaml, as well as update the kustomization.yaml to point to the configured file.    This automation was working as expected with Stable 2020.1.2. When Stable 2020.1.3 was released, deployed and tested, the configuration remained unchanged.  However, there was one pod that never reached the Running state. That pod was the sas-microanalytic-score pod.  It remained in the ImagePullBackOff status.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   After some digging we discovered that it was failing to pull the image sas-microanalytic-score (see image below).     But if you look closely you will notice that a bit earlier it successfully pulled   cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-microanalytic-score:2.42.29-20210326.1616794713240   If we take a closer look at the containers in the sas-microanalytic-score pod we see there are now two containers.   sas-microanalytic-score     viya4-mas-python-runner       But notice that the Image and Image ID for the viya4-mas-python-runner container are not fully qualified.  In fact, there is no Image ID.   Since the Image for the viya4-mas-python-runner is sas-microanalytic-score and this is the same name as the Image for the sas-microanalytic-score container, one thought was to manually replace the existing Image with the one that was fully qualified for the sas-microanalytic-score container.  Are you still with me?  Essentially, we want to replace the image for the Python runner container with the image for the microanalytic score container so that they match.     Temporary fix There are at least two ways to accomplish this.  The easiest is to edit the configuration in Lens.  Simply find the MAS pod in Lens and select it, then select the edit the configuration.       This opens the editor.  Find “image:” for the sas-microanalytic-score container and copy the image property.     Then find the property for the viya4-mas-python-runner image and change from     to     Upon clicking “Save & Close” button, the changes will be saved and applied.  And if we check the status, the pod is running.     The alternative temporary fix is to manually modify the manifest and essentially take the steps shown above, that is replace the image for the Python runner.  Then apply the manifest.  Keep in mind that the next build of the manifest will wipe out these changes. Of course, this is a test environment and the implications are minimal.  Most likely we would not want to take these steps in a production environment, thus highlighting the need for test environment, even if it is a temporary environment.     Finding the true issue and resolving it The prior steps are well and good for a one-time fix, but the same steps would have to be repeated if the manifest is applied again or if the pod was deleted it would revert to the original manifest.   After reviewing automation and other possibilities for the culprit, the next thing to look at was the original README.md for the Python transformer.  When comparing the Stable 2020.1.3 instructions with the README.md for Stable 2020.1.2, it was discovered that the following line had changed (i.e. I relied on the automation and overlooked the Deployment Notes).     Note that the instructions now indicate that the python-transformer.yaml must precede the required 'sas-bases/overlays/required/transformers.yaml'.   Up until this point the automation used to configure and deploy SAS Viya was placing the Python transformer at the bottom of the transformers block in kustomization.yaml.  It was subsequently changed to insert the Python transformer prior to the required transformers.  The manifest was then rebuilt and applied, and sure enough the image and image ID subsequently resolved properly and the pod smoothly entered a running state. Changing the order allowed the build of site.yaml to generate the Image and Image ID for the Python side car container.     How many may be lurking? This scenario raises the question about how those situations where order is important can be detected.  Well the best method is to ensure that you visit the Deployment Notes to address customizations for each version update.   If you wanted to have a quick check, a simple method to find differences is to compare the README.md files using the diff or similar command for each transformer configuration of the current and new assets, similar to what was shown in the previous section.  Of course, this requires that you keep a copy of the assets prior to applying an update.   In addition, you may want to compare the configuration files as they may have changed, and then use the most recent sas-bases configuration file for customization (i.e. don’t carry the old configuration to the new site-config).     Final Thoughts In this use case we were performing a fresh deployment of Stable 2020.1.3 but using the site-config configuration files and customization from 2020.1.2 for automation.  This isn’t really a case of updating software, but it still requires revisiting the deployment instructions and updating the configuration accordingly.   The bottom line is when performing a SAS Viya version update be sure to review the Deployment Notes and related README instructions for each existing customization.  This is especially true if you have automation built to facilitate the deployment or update process as you want to make sure you are referencing the latest version of the assets.  Failure to do so may lead you down the road of deployment failure and wasted time.   Deployment notes: SAS Help Center: SAS Viya: Deployment Notes   Find more articles from SAS Global Enablement and Learning here. ... View more

  Data scientist and Python developers who are experienced with accessing CAS from Python via JupyterHub are familiar with the steps required to launch a CAS session in a SAS Viya 3.x deployment.  Typically, this process simply requires that they specify the hostname and port as well as providing credentials in some manner.   Now that SAS Viya is deployed with Kubernetes and there are new “layers” in the deployment, there are different methods for accessing CAS from Python. If you are architecting the integration of Python or R in JupyterHub with CAS in Kubernetes, there are some considerations that may warrant your attention. In this article, we will look at several variations in which Python can communicate with CAS.   Revisiting SAS Viya 3.x Before we dig into configuring and accessing CAS in Kubernetes, let’s briefly revisit accessing SAS Viya 3.x CAS via SWAT from Python.   If you recall the process was quite simple from a user perspective.  As long as the default port 5570 was accessible from the Python session and the CAS controller hostname resolved properly, the developer or data scientist simply had to reference the CAS controller host and the port.  Of course, you had to specify credentials to authenticate, typically via an authinfo file.   If you are an installer, administrator or architect, you had to ensure the CAS controller host and configured port were accessible from the Python session (i.e.  no firewalls).  In addition, assuming CAS encryption was enabled, it was necessary to acquire the CAS certificate and make it available to all Python sessions that were expecting to connect to CAS.  And you had to ensure the SAS SWAT package was installed, along with a few other packages.   In the following diagram, you can see this environment has a dedicated host for open source programming interfaces and single machine deployment of SAS Viya.  Each of the programming interfaces establish a CAS session using the SAS Viya hostname and port 5570.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Accessing CAS in Kubernetes Now that SAS Viya is deployed in Kubernetes the way CAS services are exposed have changed. One of the key choices is whether JupyterHub will be deployed within the Kubernetes cluster in pods or outside of the cluster.   Access via a Load Balancer First, let’s take a quick look at access via a load balancer.  In the following diagram, we have a Kubernetes cluster with two SAS Viya namespaces.  Here we see a load balancer has been defined for binary communication with CAS in the Prod namespace.  The CAS client attempts to connect to the advertised load balancer IP address and port (reassigned to 5570) via the external load balancer and load balancer rules direct the traffic to the configured CAS controller.  A similar load balancer configuration could also be defined for HTTP traffic using port 8777, but the preferred method of connection is binary using port 5570.    Access via Ingress URL Yes, you can access CAS through the Ingress URL. Although this method may not be a common one it is possible.  Connecting in this manner uses the HTTP protocol and it is not as efficient as connecting via a load balancer, which provides a binary connection.  Using HTTP has another drawback and that is the HTTP protocol does not provide data message handlers and requires that the programmer provide extra data formatting.   The following example shows the Python statements required to connect to CAS using the Ingress URL. Note that we needed to provide the path the SSL_CERT_FILE and supply port 443 to ensure encrypted communication.  It is possible to define this certificate, so it is available to the launched Jupyter notebook session.     Access from JupyterHub within the K8S Cluster - Option 1 The load balancer option will be a common option for customers who have an existing JupyterHub environment and want to access a new deployment in Azure.   However, another option for accessing CAS is to deploy the Python development environment, in this case, JupyterHub, in a namespace within the same cluster as SAS Viya. This option has a couple of advantages.  First, it localizes all network traffic so it can help minimize the cost of network traffic to JupyterHub from SAS Viya. But more importantly, having JupyterHub in the cluster is likely to provide improved performance by collocating JupyterHub and Viya in the same cluster.   In the following diagram, we see JupyterHub in its own namespace and we can use the internal hostname to connect to any CAS instance within the cluster.  This eliminates the need to define an alias, as defined in the scenario with a load balancer, to provide a consistent name for the developer. The Jupyter pods can be assigned to a node pool defined specifically for non-Viya processing, or you can allow Kubernetes to assign on which nodes the pods should run.      A brief digression on hostnames In order to determine the defined hostname for each CAS controller you can use the kubectl command with a specific namespace and extract an environment variable. Of course, if the environment has an additional CAS instance or has changed the name of the default instance then the name of the queried pod will need to match.     Alternatively, we can capture the IP address by extracting it from the pod.       If you then exec into the CAS controller pod and perform a nslookup we see the following.  A close look at this output reveals that three entries equate to the client, binary and http services, if defined.  The last entry is the same as what we extracted above using the CASCONTROLLERHOST environment variable, but it includes the standard Kubernetes qualifiers of “svc.cluster.local” at the end of the name.  Both map to the same IP address and as a result either can be specified in the connection.   You will want to avoid using the hostname that is prefixed by the hyphenated IP address as this will change if the CAS server is bounced.   Here we see that we used the Kubernetes CAS controller hostname, without the svc.cluster.local qualifiers, to connect to the Stable CAS server.     And likewise, in the same Python session, we can connect to the LTS environment within our cluster using the Kubernetes hostname.     So, we see if our JupyterHub is in a dedicated namepace we can use standard hostnames available within the cluster to access CAS.  These names do not change unless the configuration is changed to modify the names.  One thing to keep in mind if going this route is that the Viya namespaces are not secured from within the cluster by default (e.g. Network Policies).     Access from JupyterHub within the K8S Cluster – Option 2 This option is a variation on the previous configuration.  Rather than deploy JupyterHub in its own namespace, you could also deploy it within the SAS Viya namespace.  Here we see that the user accesses JupyterHub via nginx and then connects directly to the CAS controller from within the SAS Viya namespace. Although only one SAS Viya namespace is shown here, it is possible to have a second SAS Viya namespace in the cluster and a second JupyterHub instance alongside the SAS Viya deployment. Of course, this adds complexity and demand on resources, but if a customer wanted to isolate JupyterHub development to a specific SAS Viya deployment this could be an option.       With this option, it is not necessary to qualify the namespace when connecting to CAS. You can simply use the standard client ClusterIP service to access CAS since JupyterHub resides within the namespace.     For example, if you create an SMP instance and change the name from “default” to “smp”, you would simply connect to to sas-cas-server-smp-client, instead of sas-cas-server-default-client shown below.      Choosing this option would have the same benefits as JupyterHub in its own namespace but would have an added small bonus of being able to directly access the necessary CAS CA certificate.    Access via NodePort Before we delve into this method it is important to note that this option is not available in AKS, as only the load balancer is available for binary access from Python external to CAS in the cluster.   This method defines a NodePort by configuring the same file as used to configure a LoadBalancer, cas-enable-external-services.yaml.  This capability allows the user to specify any of the hostnames in the cluster and the node port assigned during configuration.  Here the JupyterHub session defines the hostname for "Node 6" and port 24611 to gain access to the CAS controller. Kubernetes then routes this request via the NodePort rules to the CAS controller.       In order to determine the NodePort the following command can be used.     From the service output above we can see that the assigned NodePort is 24611 for the sas-cas-server-default-bin service. If CAS is restarted it is very likely that the assigned NodePort would change. Using this port for a quick connection check is sufficient, but if this were a long-term deployment it would be advised to set the port to a fixed value.  Remember, port 5570 is the port assigned to the CAS controller and 24611 is assigned by Kubernetes when the NodePort is created.   Here we see that when specifying one of the hostnames in the cluster and the Nodeport, 24611, we can successfully connect to CAS.     If you want to change the assigned port, you can patch the service to set the desired port. The following command will set the NodePort so that it matches the target port.  This can also be accomplished via example patch transformer included in sas-bases of the deployment assets.   kubectl patch service sas-cas-server-default-bin -n ${NS} --type='json' --patch='[{"op": "replace", "path": "/spec/ports/0/nodePort", "value":5570}]'   This concludes the variety of ways available to connect to CAS in Kubernetes, so let’s wrap up.     Final Thoughts In this article we’ve shown there are multiple ways to access CAS resources from JupyterHub, depending on where the instance of JupyterHub is deployed and how the user intends to connect.  In all examples except one, the access was via a binary connection.  However, an HTTP connection could also be considered as well. Typically, a programmer will want to want to employ the binary connection as it is more efficient than an HTTP connection.   If the JupyterHub deployment resides within the same cluster as the SAS Viya deployment, then there is no requirement to configure additional services to provide access to CAS resources.  For customers with new deployments, the addition of JupyterHub to the environment may be a consideration when planning cloud resources.   Customers that want to leverage an existing JupyterHub deployment with a new SAS Viya deployment in Azure will need to define a LoadBalancer service and ensure firewalls are open for binary access.  NodePorts can be used to access CAS resources when deployed outside of Azure (currently used by SAS internally).   Thanks for reading.   Find more articles from SAS Global Enablement and Learning here. ... View more

  Many SAS Viya users are aware of the capability to enable metrics at the CAS session level to display response time and resource consumption at the CAS action level. These metrics are displayed in the SAS log when run from a batch or compute server-initiated session. But if you wanted to capture these metrics and store them it may not be immediately obvious how to do so. In this article we look at a built-in way to capture them, but more importantly we review multiple use cases in leveraging them when deploying and managing SAS Viya environments. Built-in The builtins action set has been around since SAS Viya 3.1.  The history action within the action set has been around just as long.  However, an internet search, both internal and external to SAS, revealed very little published content about this valuable resource.  The official documentation for SAS Viya 3.5 can be found in the SAS® Viya® 3.5: System Programming Guide.   Essentially the history action allows you to “dump” the data captured about each action that has executed in the current CAS session up until the history action is executed.  The data is cumulative for the session and the history action enables the user to retrieve metrics of all actions or actions at specific offsets.   The example code shown below writes the actions recorded for the current CAS session beginning with the second-most recent action (-1) and retrieves the previous 17 actions (-18).  The current action (0) in the table is not selected as it is the running history action and data is not populated, as one would expect.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The metrics included are as follows as displayed in the following snippet from SAS Studio.     Of course, it not only contains the action executed, but also the full text of the action in the command field.  In addition, it includes the reason, severity and formatted_status fields which are populated during a failure.  Then there are the resource metrics gathered about the system and the action such as cpuUserTime, memoryOS and memory.  Some of the fields, but not all, are described in the SAS Viya 3.5 Administration: Monitoring documentation.   Here is a snippet of the data dumped by the history action on a Viya 3.5 environment. Memory fields are in bytes and time fields are in seconds. The second portion shows the formatted_status and full action command.       There is a lot of good information here but missing are timestamps and a field that provides context for the action, such as the purpose of the test.  This information would have to be added after capturing the data. Why do we care? Okay.  You might be asking yourself, Why do I care?   If you are responsible for deploying or managing Viya environments you may have wondered if changes in the environment will impact resource consumption, performance or response time.  One way to determine this is to create and maintain a suite of tests, automated of course, that run before and after changes to identify anomalies.   The test suite might be configured to access the customers data, sample datasets or both.  With a series of automated tests that mimic a customer’s workload and CAS action metrics captured to measure differences you will have a fairly efficient method to assess the impact, as well as track variations over time.  You may have a simple spot-check process in place, but imagine the comfort provided by a history of performance over time.   Let’s look at some use cases. Case 1 - Software updates Since the process of updating software on a regular basis to pick up the latest updates is fairly straight-forward with Viya 3.x, a customer desiring to identify any impact would run a suite of tests before and after software updates.  By storing the results of testing prior to software updates and then storing the results post-updates, the user has a measuring stick to determine if the software update warrants further attention. Case 2 - Adding/removing CAS worker nodes A customer may want to add or remove a worker node from the CAS server for a variety of reasons.  Maintaining and running a test suite and capturing the results before and after changes should provide some indication as to what the user will experience.  One would expect adding a worker node would improve response time and if this was not the case, it would be easy to identify response times and resources by CAS action. Case 3 - Configuration changes Using a test suite and capturing the CAS history could also be used if a customer wants to tweak their CAS configuration.  Take for example one of our favorite tuning knobs, CAS_DISK_CACHE.  If a customer decides they want to modify the number of directories for CAS_DISK_CACHE or place them on a faster device, capturing data from their own CAS actions would provide a good indication of any performance gains. Case 4 - Moving from on-premise to the cloud One key reason customers consider moving to the cloud is to manage costs.  However, if they over-provision the move could be more costly than they anticipated.  As more and more SAS customers move to the cloud, running a test suite in both environments and capturing the history could provide some insight into whether they are over-provisioned or not.  In addition, it may highlight other potential issues with the CAS configuration. Case 5 - Moving from SAS Viya 3 to SAS Viya 4 Any existing customer that is currently running SAS Viya 3.x and plans on moving to the upcoming SAS Viya 4  release could take these steps to better understand and gain insight as to how the two environments compare.  For example, if one is significantly slower than the other, is it slower for all CAS actions?  Or is it just some actions? Case 6 - Canary testing User usage patterns and data in customer environments change regularly.  As a result, another use case would involve scheduling a test and keeping a record of the history results.  This could be done during a window that is fairly idle to minimize variation.  The purpose is similar to the other use cases and that is to quickly identify changes, outliers and anomalies, and ultimately pinpoint where to dig further. Case 7 - Fresh deployment (SAS Viya 3 or 4) Finally, establishing a test suite for fresh deployments would provide a baseline for the new environment. This use case essentially is the first phase of the canary testing use case.  If we have a baseline then the installer has a sense of resource consumption and response time from the very beginning of the deployment. Automation Of course, we want to establish a test and capture the data as quickly as possible.  Therefore, it is imperative to automate as much as possible. Storing the test artifacts such as scripts and code in Git provides an easy method of retrieving the items required.  Jenkins or a similar tool could be used to automate the actual tests and provide a tracking mechanism for each test. Using these tools to streamline the testing and data capture would apply to all of the above use cases, but it would be especially true of the canary testing as testing is an ongoing process. Limitations As far as I could determine capturing this data using the history CAS action of the builtins action set is only available from a programming interface such as SAS Studio, batch or open source programming languages such as Python (I didn’t test open source but this should work as well).  As a result, the sessions initiated by other SAS Viya applications would not be accessible.   Another obvious limitation is this only captures CAS actions.  There are many other components to SAS Viya that are not addressed.  But since CAS is the primary analytic engine for most SAS Viya solutions it is certainly a good place to start.   In some of the use cases listed above it is implied that there are two environments available for testing.  This may not be the case.   And finally, CAS action testing would provide greater consistency if run during idle periods.  This may not be possible, depending on the customer’s workload. Final Thoughts The CAS builtins.history action is a valuable capability that unveils an easy way to capture CAS action resource and response time metrics.  Once captured this data could subsequently be saved in a table to provide context for comparison. The preserved table can then be used for multiple use cases to compare variations in resource consumption or response time and detect anomalies.  The goal is to ultimately provide a more reliable and performant environment. Automating the testing, data capture and comparison will help provide quick results. Just keep in mind a few limitations exist.   If you’ve done anything similar to this feel free to share in the comments below.   Thanks for reading. ... View more

  Most likely you've heard the term Jenkins, but maybe you haven't had the time to dig into the ways it may benefit you as a SAS Viya installer, architect, administrator, or user. In brief, Jenkins is an open-source automation tool used heavily in software development. However, its capabilities are wide-ranging and can certainly be applied to many facets of a SAS Viya 3.5 deployment.   One of the commonly-used features of Jenkins is its ability to trigger builds based on an event or a schedule. For example, when configured to interact with Gitlab, you can trigger a build to execute each time an update to a Gitlab project occurs.  Assuming that example, let's take a look at what is required to trigger a Jenkins build and review some potential use cases. Jenkins Jenkins is an open-source automation server with many available plugins to extend functionality.  It is used by R&D and testing to automate building, deploying,  and testing throughout  the software development life cycle.   But don't be fooled into thinking that it is strictly for software development. There are many capabilities that enable it to be an effective tool to automate many SAS Viya 3.5 processes, both deployment and operations related.   If you've ever seen a screenshot or have logged on to the application, don't let the "classic" web interface fool you.  Behind that interface is a ton of functionality to automate your most-loved processes.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   If you want to modernize the interface, you can check out the Blue Ocean interface. Check out the online doc for more details.   Let's review some key Jenkins terminology.   Plugins - Much of that functionality is the result of the plugins that interface with other technologies (e.g. Gitlab, Github, cloud providers, etc).  And lest I forget, it is extensible.  So you can enrich the functionality ever further if you wish by writing your own plugin.   Master - The central coordinating process/server which stores the configuration, provides and manages plugins, and renders the interface.   Project (Job) - At one point these terms were used interchangeably. However, the term "Job" has been deprecated.  Essentially projects are the core runnable tasks that are controlled and monitored by Jenkins.  Projects are created using the "New Item" menu and after selecting a type of project, you are presented with a series of configuration sections. The project can represent a single step or a complex combination of steps that run across multiple machines.   Build - A build is simply an execution or run of a project.   Artifact - Artifacts are files or objects created during a build.  Artifacts can be tossed at the end of a run or saved for future use.   Node - Nodes are the hosts/machines that are set up to run the project builds when run from a master.  In order for it to run tasks on a node, an agent must be defined on that machine.  After a node is registered to the master, the master then distributes work to the nodes.  Work can be directed using a variety of options.  The Master is also a node.   Workspace - This is a disposable directory on the node that is used as a working directory for the build.   Pipeline - A user-defined model of a continuous delivery pipeline.  Okay, that definition may not be too helpful.  A pipeline is a definition of a series of stages to be executed either serially or in parallel on one or more hosts. Stages may contain one or more steps.  For you mainframers, it reminds me of an uber-functional JCL with a web interface.  But I digress…   Stage - a grouping in a Pipeline that defines a subset of the entire pipeline   Step - a single task within a pipeline; a group of steps can be used to define a stage.  An example would be a single host command   For more terms, check out the Jenkins glossary.   Here is a simple architecture diagram of the Jenkins architecture (thanks Rob Collum).   Notice we can have agents on a variety of platforms and each agent needs a workspace to store its repositories and project content (artifacts).  If running a parallel pipeline, all three agents can be actively working at the same time.  Logs for each stage are made available to and stored on the Jenkins server (Master).  The Master can integrate with a variety of Git flavors as well as other software technologies.     In this example, we take advantage of Jenkins' Docker image.  Using Docker to initiate a Jenkins environment provides a fast path to quickly getting access to a Jenkins server. Gitlab I'll keep this section short.   Most likely you've dealt with one or more of the Git flavors in recent years.  Gitlab, and its competitors, are web-based DevOps lifecycle tools which provide a Git-repository manager for versioning, wiki capabilities, and issue-tracking along with various CI/CD pipeline features.  Gitlab operates using an open-source license.  A fair amount of overlap exists with the CI/CD pipeline features and Jenkins.  In this example, we use Gitlab purely as a versioning tool. Triggers A key feature in building automation in Jenkins is the ability to automatically kick-off (trigger) a project build as a result of an event, a request, or a predefined schedule.  One manifestation of this ability is configuring a Jenkins project to perform a task when a push to a Gitlab project is issued and completed.  For example, you have a YAML file that you are modifying in Gitlab and each time you modify it you want to perform syntax checker on the updated file (could be JSON, XML, etc). This is a simple example, but the point is to drive home the idea behind the example.  So let's walk through the steps to enable this ability. Jenkins Key Components In order to create the trigger build we need to ensure we have the required software so that Jenkins can integrate with a Gitlab repository. Jenkins software - probably the fastest way to get Jenkins installed and running is via a Docker image.  If you have Docker installed you could have Jenkins running within minutes.  Images can be found here. Gitlab instance - In many cases, this may already be available, you will just need to acquire an account that creates a repository or request authorization.  If a Gitlab instance isn't available, using Docker to deploy Gitlab is a fast way to get an instance running for your testing. Jenkins Plugins for Gitlab - Since in this example we will be interacting with Gitlab, there are specific plugins required.  If you use the standard Docker image, you will need two additional plugins.  To add plugins, go to Jenkins home => Manage Jenkins => Manage Plugins and then select the Available tab.     Key plugins are Gitlab Plugin and Gitlab Hook Plugin.  If you want to build merge requests from Jenkins, include the Gitlab Merge Request Builder. Jenkins may have to restart to enable the plugins.  This takes a couple of minutes.  They will then show under the Installed tab   At this point you should have the key software pieces in place to begin configuration to perform a triggered build. Trigger Project Configuration and Execution To activate a triggered build, we will break this up into two phases, configuration and operation.   The following diagram highlights the sequence of actions required to configure a Jenkins project to be triggered when an update is pushed/merged/committed to a Gitlab project.  There are four major steps.     We first need to generate a personal Gitlab API Access Token that will be used by Jenkins to access the associated Gitlab project.  The access token is acquired via the Gitlab user's settings.  Once the token is generated, it should be added to Jenkins' credentials. Next, we want to create a new project in Jenkins.  This can be a freestyle or pipeline project.  Within the project, we define what type of trigger we should initiate in the Jenkins job.  When the option, Build when a change is pushed to GitLab, is selected it provides the Gitlab webhook URL to be used in configuring Gitlab.  The webhook will look something like http://myjenkins.sas.com:8080/project/MyTriggerProject (not a real URL).  Note the other available triggers.     With the Jenkins Webhook URL return to Gitlab with the desired repository selected and select Settings => Webhooks.  Here you add the URL and define which events in Gitlab you want to create the trigger.  Trigger examples are Push, Merge, Comments, etc.  Once the webhook is added, you can test it to ensure it triggers as expected. Now that we have the webhook configured in Gitlab, return to Jenkins and configure your Jenkins project.  There are many possibilities here.  Here we will present a very simplistic example with a few of the options. Define on which node you want the project to execute.  In the example, we execute on Node01. Add the Gitlab repository to the Jenkins project. This repository will then be cloned to the workspace of Node01. We defined the trigger in step 2 Define what type of build action we want to execute when the trigger is initiated.  Some examples are to execute a script that is part of the repository or run a system process against a file or multiple files in the repository. At this point, the configuration is done and we are ready to begin testing (operation).     Depending on which Gitlab actions were configured for the Webhook, you want to generate that action.  For example, make a change to a file in the repository and commit the update which results in a Push. The Gitlab action creates a call to the Webhook, which calls Jenkins with the trigger project URL. The trigger build is initiated on the server, but the actual build was specified to run on Node01. So the required pieces from project are copied to the agent node to the defined workspace location. Since we specified a Gitlab repository in the project, the build information requests that the Gitlab repository be cloned to the agent node. With all the pieces in place, the build process executes on Node01 and the results and logs are returned to the server. So there you have a simplistic view of the steps to configure Jenkins and Gitlab to initiate a trigger build job. A Few Trigger Use Cases Earlier we mentioned a very simple use case.  Assume we have the SAS Viya playbook stored in a Gitlab repository and we are planning a new deployment.  Most likely you will be making changes to the vars.yml file.  As you are making changes you could define a trigger build that will invoke a YAML syntax checker, such as YAMLlint, each time you update vars.yml to validate it.  This could save you time down the road with debugging changes to your deployment.   One key phase of a new deployment, or changes to an existing deployment, is to validate that services in a new or updated environment are functioning as expected. A trigger build could be established to be initiated after all services are running to verify the health of services and subsequently perform basic functionality testing.  Jenkins can be configured to retain the logs and output.  It captures response times as well. As a result the Jenkins user interface can provide a historical view of past validation tests and you can quickly identify executions that do not complete within the expected execution time.   Beyond these, it is up to your imagination to identify the many possibilities of creating trigger builds for a SAS Viya 3.5 deployment. Final Thoughts In this article, we've reviewed the basic process flow of triggering a Jenkins project build.  If you haven't been exposed to Jenkins before and your responsibilities include performing repetitive processes that would benefit from automation it may be worth your time to take a look at its capabilities.  With Gitlab and Jenkins Docker images you can fairly quickly have an environment running and begin trigger testing.   Thanks for reading! ... View more

The "openness" of SAS Viya has been advertised and touted for the last couple of years.  Many, if not most SAS customers have skills and investments in open source resources that they want to exploit and leverage to their business advantage.  To a certain extent, SAS delivered some of this capability prior to SAS Viya.  Three years ago I wrote an article on deploying SWAT for Python and showed how it opened the door to CAS. But the capabilities of SAS Viya have grown significantly since that blog was written.   Along the way you may have heard about an integration point or some slick functionality with regard to interfacing with SAS software, but you may not be aware of the many ways SAS Viya can interact with various open source technologies.  In this article we'll try to capture many of the touchpoints and catalog them to provide a quick reference (sorry no pictures).  In a subsequent article we'll look at some of the more common scenarios from an architecture and topology perspective. Open Source > SAS and SAS > Open Source Let's state the obvious.  The integration with open source software is a two-way street.   For example, a user working with R may want to transfer some local data from their environment to CAS and subsequently join with data already loaded.  She then executes models in CAS followed by returning the results to their R environment.  Therefore she is reaching out to SAS resources from her open-source programming environment.   Alternatively, a modeler may be working with SAS Model Studio (VDMML) and have a block of Python code that he wants to invoke as part of their modeling scenario.  In this scenario he is exploiting the ability to integrate Python code into their VDMML project.   These are just two examples of interactions, but from an architect’s perspective, it may be useful to be aware of the many ways SAS Viya interacts with open-source software.   We'll use the following categories to group the open capabilities. Open-source applications leveraging SAS execution engines SAS applications leveraging open-source execution engines Application development/integration Data integration Open-source applications leveraging SAS execution engines Python Let's start with the most popular language used to access SAS Viya and CAS.  There are multiple touchpoints when interacting with SAS Viya and CAS.   Getting Started with SAS Viya for Python (developer.sas.com) SWAT (Scripting Wrapper for Analytics Transfer) - The most obvious package, and probably the most popular, is SWAT. Once installed this plugin opens the Python doors to access the powerful capabilities of CAS. Getting Started with SAS Viya for Python (documentation.sas.com) Python SWAT (github.com) saspy - This package allows a programmer to connect to and submit code to a SAS session. There are numerous connection methods, but only one (HTTP) that is specific to launching a Compute Server on SAS Viya.  The remaining methods are primarily used to connect to SAS 9.4 sessions. saspy Interface (github.com) DLPy - SAS Deep Learning Python package provides high-level Python APIs to deep learning methods in SAS VDMML. DLPy Interface (github.com) ESPPy - The ESPPy package provides the capability to create SAS ESP models via programming in Python. It can connect to a running ESP server and interact with running projects, continuous queries, windows and various other components of the ESP environment. ESPPy Interface (github.com) sasoptpy - The sasoptpy package introduces a modeling interface to SAS Viya Optimization and SAS/OR solvers. It provides a quick way for users to deploy optimization models and solve via CAS actions. sasoptpy Interface (github.com) sas-kernel - The sas-kernel package for Jupyter Notebooks opens the data manipulation and analytics capabilities of SAS within the notebook interface. This interface can be used to execute SAS code and view results inline. sas_kernel Interface (github.com) R SWAT - Similar to Python when the SWAT module is installed a user can be developing R programs and call CAS actions directly from their programming environment. Getting Started with SAS Viya for R (developer.sas.com) Getting Started with SAS Viya for R (documentation.sas.com) SAS Viya and CAS REST APIs The REST APIs provide a mechanism to access both Viya and CAS services from many different programming languages that provide REST call functionality. SAS Viya and CAS REST APIs (developer.sas.com) Java The SAS Java Client interface is the equivalent of the SWAT package for Python and R. It enables Java programmers to access CAS resources. This includes loading data and running CAS actions against that data loaded to CAS. Getting Started with SAS Viya for Java (developer.sas.com) Getting Started with SAS Viya for Java (documentation.sas.com) Lua  SWAT - As we saw with Python and R, SWAT for Lua enables the programmer to unleash the power of CAS from their Lua programs. Getting Started with SAS Viya for Lua (developer.sas.com) Getting Started with SAS Viya for Lua (documentation.sas.com) SAS applications leveraging open-source execution engines SAS VDMML - Python and R SAS Model Studio is the interface used to create, execute and evaluate models in SAS VDMML. There are numerous nodes available for predictive analytics when working with a project.  One of those nodes is the Open Source Code This node enables the user to seamlessly insert R or Python code into the project to provide complementary analytics processing VDMML - Open Source Code (documentation.sas.com) SAS Visual Forecasting - External Languages Package The External Languages Package was released with SAS Visual Forecasting on SAS Viya 3.5 to provide SAS > (R, Python) > SAS roundtrip communication while running in distributed mode across CAS nodes. Running in distributed mode provides scalability to Python and R. It handles all data communication from CAS tables to Python or R and back to CAS External Languages Package (documentation.sas.com) SAS Model Manager - PMML Support PMML is an XML markup language developed to support exchanging models between modeling systems and scoring platforms. SAS Model Manager supports importing a majority of standard-compliant PMML models and scoring them within a SAS environment using the SAS PSCORE procedure. PMML Support (documentation.sas.com) SAS Model Manager - PyMAS SAS Model Manager enables a customer to store models in a common repository, as well as organize them within projects and folders. The PyMAS package supports models that contain a custom Python code file.  This package is delivered during the installation, but must be enabled via configuration. SAS Model Manager - PyMAS (documentation.sas.com) SAS Intelligent Decisioning - PyMAS SAS Intelligent Decisioning provides decision management capabilities to automate the process. If a customer has existing Python code files or writes new code for decisions, these can be executed as part of a decision flow by loading the code to the SAS Micro Analytic Service.  The PyMAS package provides this ability to load and execute Python code. SAS Intelligent Decisioning - PyMAS (documentation.sas.com) SAS Event Stream Processing - PyMAS SAS ESP can use the SAS Micro Analytic Service to execute a block of Python code as part of a running ESP model. That block of code, defined at the project level, can contain functions written Python. SAS Event Stream Processing - PyMAS (documentation.sas.com) Application development and integration There are numerous statements, PROCs, functions and SAS programming features that interact with open-source resources.  In this section, we will touch on a few of them. SAS Event Stream Processing SAS ESP Connectors and Adapters - SAS ESP ships with many adapters and connectors, the data source/target interfaces, that retrieve data and load data to many common open-source data sources. A few examples are Kafka, Cassandra, HDFS, and RabbitMQ. SAS ESP Connectors and Adapters (documentation.sas.com) WebSocket API - The WebSocket API enables two-way communication between a client written by a developer and a remote host that accepts that communication. In the ESP context, the ESP server would be the remote host server and the languages invoking web sockets could be written in Java, JavaScript, C++, Python and IOS. SAS ESP WebSocket API (documentation.sas.com) JavaScript API - The ESP JS API can be used to create and manage ESP projects in an ESP server from within web pages on platforms independent of the ESP server. SAS ESP JavaScript API (documentation.sas.com) Publish/Subscribe API - SAS ESP provides publish/subscribe APIs for C, Java, and Python.  These APIs can be used to publish event streams into a Source window of a running project as well as subscribe to windows within a project to retrieve events. SAS ESP PubSub API (documentation.sas.com) Apache Camel - Apache Camel is a framework that enables a user to integrate applications into a single, cohesive architecture to consume and produce data. SAS ESP Apache Camel (documentation.sas.com) Apache Nifi - Apache Nifi is a system used to automate data flows between multiple systems. SAS ESP Apache Nifi (documentation.sas.com) Mobile SDKs (IOS and Android) Available Software Developer Kits used to create mobile apps accessing SAS Visual Analytics. Mobile SDKs (developer.sas.com) Visual Analytics SDKs The Visual Analytics SDKs enables web developers to use a collection of JavaScript libraries to embed SAS Visual Analytics content with custom web pages and web apps VA SDKs (developer.sas.com) PROC FCMP The SAS Function Compiler (FCMP) procedure enables a programmer to create, test and store SAS functions, CALL routines, and subroutines that can then be invoked in a DATA step or SAS procedure. These user-created functions can subsequently call Python functions, allowing a customer to leverage Python functions they may have already created and stored.  The Micro Analytic Service provides the execution environment for Python functions Using PROC FCMP Python Objects (documentation.sas.com) FCMP Procedure (documentation.sas.com) What’s New in FCMP for SAS 9.4 and SAS Viya (www.sas.com) Git functions There are many Git functions available in SAS programming that enable a developer to access and manage Git projects from SAS code. They include many of the common actions taken with a Git project. Using Git Functions in SAS (documentation.sas.com) Hadoop Although most customer Hadoop deployments are commercial releases, the basis of these deployments is built on open-source software. PROC HADOOP can be used to submit HDFS and MapReduce commands to Hadoop instances from a SAS program. HADOOP Procedure (documentation.sas.com) PROC GROOVY enables programmers to push Groovy statements as uncompiled code to a JVM. GROOVY Procedure (documentation.sas.com) Others The FILENAME statement provides access to many data sources, of which there are numerous open-source sources. PROC HTTP issues HTTP requests to a specified server and is available in SAS 9 and SAS Viya. HTTP Procedure (documentation.sas.com) PROC LUA allows a programmer to run LUA statements with SAS code. LUA Procedure (documentation.sas.com) The X statement provides the capability to execute an operating-environment command from within a SAS session. X Statement (documentation.sas.com) Data integration Like SAS application integration points, there are numerous open-source data sources that can be accessed by SAS Viya. SAS/ACCESS Interfaces and SAS Data Connectors The ACCESS interfaces and data connectors provide access to open data sources from SAS compute and CAS sessions. Example interface and connectors are MySQL, Spark, and PostgreSQL. SAS/ACCESS for Relational Databases (documentation.sas.com) Configure Data Access for SAS Viya (documentation.sas.com) Support for Databases (support.sas.com) SAS Cloud Data Exchange SAS Cloud Data Exchange provides data connection capability for SAS Data Preparation. The connection provides secure data transfer from an on-premises data store to cloud-based instances of SAS Viya, enabling SAS Viya applications to access to those data sources.  The SAS Data Agent is deployed on-premises and configured to transfer data to and from SAS Viya.  It is capable of using many of the interfaces and connectors noted above. Cloud Data Exchange and SAS Data Agent (documentation.sas.com) SAS Data Agent 2.5 for Linux: Deployment Guide (documentation.sas.com) A quick look at SAS 9 While digging into this topic I came across this presentation that highlights many of the basics when looking at SAS 9 and open source.  To become familiar with some of those integration points navigate here to check it out: Open Source and SAS 9 (support.sas.com). Wrapping up As you can there are many touchpoints regarding SAS Viya and open-source software.  This list is not a comprehensive list but has been compiled to raise awareness.  Now that we are aware of many of the integration components it begs the question for architects of what software is needed and where should it be placed. We'll cover some of those scenarios in a future article.   My thanks to Simon Williams for his contributions to this article.   Thanks for reading.     ... View more

A highly available SAS ESP architecture consists of 1+N machines that host the SAS ESP server where the ESP project(s) runs. If a failover occurs it is important to bring the failed instance back online as soon as possible. This is especially true if there are only two ESP instances configured (possible but not recommended). Bringing the failed instance back in sync with the existing instances is not an overly-complex exercise but it requires an understanding of key components and steps that need to be taken.   For greater detail on failover, read ESP Failover using Kafka, which explains the components and steps for ESP failover using Kafka as the message bus. In this article, we will review those components and actions required to restore a failed ESP server and bring it in-sync with other running servers.   First a Recap Before we dig into the restoration process, let's quickly review the failover architecture using Kafka as the message bus.   In the following architecture there are three ESP servers running the same model. There is one active instance and two standby instances. However all are running in-sync in terms of processing events/messages. The active instance is the only instance that is feeding events/messages to the Kafka topic via the subscribe connector and adapter(s). The message ID used to keep track of which event has been processed is managed via ZooKeeper and the subscribe connector. For more in-depth insight, check out the Failover using Kafka article noted above.   Select any image to see a larger version.   Failure If the active instance fails one of the two remaining standby instances will resume processing. The subscribe connector on the instance transitioning to active knows at which event it should resume processing as ZooKeeper is tracking the message IDs. That standby instance then becomes the active instance.   Persisting and restoring the failed instance Maintaining high availability requires bringing the failed instance back online and in-sync with the remaining instances as soon as possible. But how is this accomplished?   Persisting The persist process is the means of saving the state of a running server, either active or standby. This process will save the state of the server to disk at the point it was taken. There are several steps that need to be taken to capture the state. Stop the running project so no events are flowing through the model Persist the state to disk Start the project to resume processing Start the connectors within the project (the connectors are not started when the project is started) Each of these steps can be achieved using the ESP client. Here are examples of using the ESP client using the HTTP protocol to accomplish the steps above.   # Stop the running project dfesp_xml_client -url “http://intesp04.****.***.com:5556/SASESP/projects/dstatkafka/state?value=stopped” -put # Persist the project to directory /opt/sas/dstat/failoverdfesp_xml_client -url “http://intesp04.****.***.com:5556/SASESP/projects/dstatkafka/state?value=persisted&path=/opt/sas/dstat/failover/” -put # Start the project dfesp_xml_client -url “http://intesp04.****.***.com:5556/SASESP/projects/dstatkafka/state?value=running” -put # Start the connectors (they are not started when the project is started) dfesp_xml_client -url “http://intesp04.****.***.com:5556/SASESP/projects/dstatkafka/state?value=connectorsStarted” -put   After the connectors have been started the project will be resumed. It may take a few minutes to become fully in sync with the other ESP servers. If the above statements are scripted the output from each of the commands would look similar to the following.   <response> <message>project dstatkafka successfully stopped</message> </response> <response> <message> <response> <message>project dstatkafka successfully saved to /opt/sas/dstat/failover/</message> </response> </message> </response> <response> <message>project dstatkafka successfully started</message> </response> <response> <message>connectors started for project 'dstatkafka'</message> </response>   So what does a persisted directory structure look like? Here is a screen shot of a persisted directory. Everything under the failover directory is created by the persist request.     In this case the size of the directory is 56MB. Obviously this is relatively small but subject to change over time depending on retention policies.     This is something to keep in mind as the larger the model and data retained in the model, the longer it will take and more disk space it will require to store the state.   There are two methods of storing the current state. Persist the project state locally for each ESP server Persist the project state to shared storage from a standby server (e.g. NFS mounted file system) Persist locally If you persist the state locally for each ESP server it requires that each project be stopped, persisted and restarted. As you can imagine performing these steps on the Active instance will create a "pause" in processing events. The length of this pause will be dependent upon the size of the data persisted and the speed of the storage. This is not ideal as real-time systems will be impacted by the pause. And as you can imagine this should be on a storage that is not susceptible to failure.   This diagram shows each instance persisting the project to local storage.   Note that this diagram does not represent the testing environment referenced throughout the article.   Persist to shared storage As noted above the alternative is to persist the ESP project from a single standby server to storage shared by each instance. This method has a couple of benefits. First there will only be one copy of the persisted store and each instance can restore from that copy. In addition there is no need to pause the active instance. The one small downside of this method is that automation will need to be built to track the active and standby instances to ensure the project is persisted from a standby instance.   This diagram shows a standby instance persisting the state of a standby instance to shared storage. Although not shown here all three machines have read/write access to the NFS mounted file system.   Note that this diagram does not represent the testing environment referenced throughout the article.   Scheduling persistence In most deployments the most effective way to ensure that an ESP server is persisted is to schedule it via cron or some other scheduling tool. The commands to persist the state of a standby instance shown earlier can be converted to a simple script. This can be scheduled as needed. Depending on the amount of data processed and system requirements it may be necessary to schedule it multiple times per day. This will reduce the amount of "recovery" time to be in sync with the other instances.   Restoring a persisted project When an ESP server failure occurs it must be restarted and the project loaded and started to bring the server in-sync with the running instances. Here are the steps for restoring a persisted project. Start the XML server without loading a project Load the project but don't start it Restore the project from the persisted state Start the project Start the connectors Like the persist process all but the first step can be executed using the ESP client. Here are command line statements for each of the steps.   # On intesp03 as sasesp # Start the ESP XML server but do not load a project nohup dfesp_xml_server -pubsub 5555 -http 5556 -badevents ~/dstat_badevents.txt > /opt/sas/dstat/intesp03_dstat_`date +%d-%m-%y.%H.%M.%S`.log & # Load the project but do not start it dfesp_xml_client -url “http://intesp03.****.***.com:5556/SASESP/projects/dstatkafka?start=false” -put “file:///home/sasesp/dstatkafka.xml” # Restore the state of the model from the persisted state dfesp_xml_client -url “http://intesp03.****.***.com:5556/SASESP/projects/dstatkafka/state?value=restored&path=/opt/sas/dstat/failover/” -put # Start the project dfesp_xml_client -url “http://intesp03.****.***.com:5556/SASESP/projects/dstatkafka/state?value=running” -put # Start the associated connectors dfesp_xml_client -url “http://intesp03.****.***.com:5556/SASESP/projects/dstatkafka/state?value=connectorsStarted” -put   If successful these commands will generate the following responses.   <message>load project dstatkafka succeeded</message> <response> <message>project 'dstatkafka' successfully restored from '/opt/sas/dstat/failover/'</message> </response> <response> <message>project dstatkafka successfully started</message> </response> <response> <message>connectors started for project 'dstatkafka'</message> </response>   Upon reviewing the log from the ESP XML server we can see the progress from above statements. After the project is loaded, restored and started, the server is switched to Standby state but no processing activity is started until the connectors are started. Once the connectors are started processing of events stored in the Kafka topic resumes and communication with ZooKeeper is established allowing it to sync with the other ESP servers.   2019-01-17T13:22:37,289; INFO ; 00000004; DF.ESP; (dfESPengine.cpp:762); [Engine0007] dfESPengine::initialize() dfESPengine version 5.2 completed initialization 2019-01-17T13:22:37,292; INFO ; 00000004; DF.ESP; (Esp.cpp:668); [XMLServer0001] esp engine started, version 5.2, pubsub: 5555 2019-01-17T13:22:37,292; INFO ; 00000004; DF.ESP; (Esp.cpp:791); [XMLServer0001] starting esp server   2019-01-17T13:22:37,292; INFO ; 00000010; DF.ESP; (Http.cpp:602); [XMLServer0001] starting HTTP server on port 5556, type=admin 2019-01-17T13:25:19,990; INFO ; 00000013; DF.ESP; (dfESPconnector.cpp:1378); [Connectors0087] dfESPconnector::setState(): Connector in group default has new state: stopped 2019-01-17T13:25:19,992; INFO ; 00000013; DF.ESP; (C_dfESPpubsubApi.cpp:406); [PubSub0056] C_dfESPpubsubInit(): Successfully loaded pubsub plugin libesp_native_ppi-5.2.1.so from /opt/sas/viya/home/SASEventStreamProcessingEngine/5.2/lib/plugins/ 2019-01-17T13:25:20,065; INFO ; 00000015; DF.ESP; (dfESPwindow_aggregate.cpp:209); [Aggregate0017] dfESPwindow_aggregate::postInputFinalizer() for window Aggregate1: Aggregate window using additive optimizations 2019-01-17T13:25:20,526; INFO ; 00000016; DF.ESP; (dfESPproject.cpp:790); [Project0009] dfESPproject::startPubSub() for project _meta_: Pub/Sub services are enabled for port 41241, encryption disabled, client authentication disabled 2019-01-17T13:25:20,526; INFO ; 00000016; DF.ESP; (dfESPproject.cpp:632); [Project0016] dfESPproject::finalizeProject() for project dstatkafka: Returned immediately, project was already finalized 2019-01-17T13:25:20,528; INFO ; 00000016; DF.ESP; (dfESPproject.cpp:790); [Project0009] dfESPproject::startPubSub() for project dstatkafka: Pub/Sub services are enabled for port 39038, encryption disabled, client authentication disabled 2019-01-17T13:25:20,601; INFO ; 00000029; DF.ESP; (dfESPkafkaConnector.cpp:2004); [Connectors0049] dfESPkafkaConnector::start(): Hot failover state switched to standby 2019-01-17T13:25:20,602; INFO ; 00000029; DF.ESP; (dfESPconnector.cpp:1378); [Connectors0087] dfESPconnector::setState(): Connector subkafka in group default has new state: running 2019-01-17T13:25:20,604; INFO ; 00000029; DF.ESP; (dfESPconnector.cpp:1378); [Connectors0087] dfESPconnector::setState(): Connector pubkafka in group default has new state: running 2019-01-17T13:25:20,607; INFO ; 00000036; DF.ESP; (dfESPkafkaConnector.cpp:294); [Connectors0098] dfESPkafkaConnector::z_watcher(): Zookeeper watcher event: watcher = SESSION_EVENT, state = CONNECTED_STATE, path = 2019-01-17T13:25:20,609; INFO ; 00000037; DF.ESP; (dfESPkafkaConnector.cpp:331); [Connectors0097] dfESPkafkaConnector::z_watcher(): Created zookeeper node /ESP/server-n_0000000132 2019-01-17T13:25:20,611; INFO ; 00000038; DF.ESP; (dfESPkafkaConnector.cpp:275); [Connectors0096] dfESPkafkaConnector::watchActiveZnode(): Watching zookeeper node /ESP/server-n_0000000131 2019-01-17T13:25:30,894; INFO ; 00000035; DF.ESP; (dfESPclientObj.cpp:585); [PubSub0081] dfESPclientObj::doRateProcessing(): Current publish rate: 3912.458950 events/sec   In this example the persisted store was placed on an NFS-mounted drive. The state was persisted from host intesp04, which was the standby instance before the failure. After the active instance failed on intesp03 the XML server was restarted on intesp03 and the persisted state created from intesp04 was used to restore the state on the newly started instance on intesp03.   Here we see a restored standby instance on the machine where the ESP server failure occurred (sasespsrv01). The restore process used a persisted store on an NFS mounted file system to sync up the ESP server on the first machine. The second machine (sasespsrv02) has become the active instance.   Note that this diagram does not represent the testing environment referenced throughout the article.   Understanding the restore process timeline Now that the steps for persisting and restoring an ESP server have been outlined, let's take a look at the timeline of events. If we consider the events/messages that are appended to a Kafka topic, those events/messages are retained for a period of time as defined by Kafka system settings. The time window for those events must precede the cycle for the schedule persist. Let's attempt to illustrate this.   The following time line shows a continuous three-day flow. The scheduled persist appears to run early in the morning and the retention within the windows of the project typically contain data for more than a day. The retention for Kafka also appears to keep events for slightly more than a day. But the key is the events in Kafka precede the time when the scheduled persist occurs.     So what happens when a failure occurs? In the following time line we see a failure occurs around noon. But the restore from the persisted state does not occur until evening. This is OK because the restore will be applied to a standby ESP server instance and the events/messages in Kafka with which it needs to sync precede the scheduled persist time. The initial state of the restored instance will be in the state when it was persisted and the events/messages required to sync will be identified in Kafka and processed until in-sync with the other ESP servers.     A noteworthy item During persist and restore testing I encountered an issue with the ESP 5.2 Kafka connector. After persisting a standby instance, and subsequently starting the project and connectors so that it was processing again, the standby instance failed to transition to active state when the initial active instance was terminated. The issue was identified and will be fixed in ESP 6.1.   Final Thoughts So there you have it. A way to store the state of an ESP server and then use that stored state to restore the state of a failed server instance. Hope it was of some value to you.   My thanks to Andy Tracy and Andrey Matveenko for their contributions to and input on this article. ... View more

The SAS GEL team manages numerous collections of test VMs that are heavily used by a large number of workshops. For a majority of the workshops each student is assigned his or her own collection. However, the larger collections are shared in order to minimize demand on the test environment and ensure there are enough machines for other workshops. The shared environments are long-running reservations that many students use over a period of months for multiple workshops. It was on a shared reservation where a memory-related issue surfaced that I believe is worth sharing. The Testing Environment The shared testing environment that is used by multiple workshops at SAS is configured as follows. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   When students enroll in a class, they receive a client machine that will connect them to the environment. Over the course of a couple of months several hundred users might connect to the shared resource for the virtual learning environment (VLE). As a result the memory usage of the environment experiences growth over time. Lack of Memory? Recently a student sent a notification to the VLE message board that they were stepping through a hands-on exercise but that SAS Studio 5 couldn't launch a compute server. You may have seen this message before. However, the cause behind the failure is not immediately obvious:     Upon digging into this message I came across an internal SAS note that provides diagnostic steps. After sifting through several logs, I discovered the following message in the consul log: 2019/04/15 11:08:13 [WARN] agent: Check "memory-check" is now warning   Based on this, it appeared to be a memory issue.   Looking at the amount of free memory it shows that 85 GB out of 94 GB are in use (~90%). There are two GB available and 8 GB used by buffer/cache, but no free memory. The man page for the free command defines used memory as [used = total - free - buffers - cache].   (The numbers shown here do not add up accurately because of rounding to gigabytes.)     The key field here is the available field, which is defined as "Estimation of how much memory is available for starting new applications, without swapping." So it appears that only 2 GB are available to start new applications. Based on the limited available memory it seemed worth the time to attempt to free some memory. The top Command The Linux "top" command is an efficient tool that can be used to quickly find the largest consumers of memory. You can use the sort option within "top" to sort entries on the RSS (Resident Set Size) column to find the largest consumers. A review of the sorted output reveals there were five java processes using over 5 GB of RAM: Top Memory Users   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  4659 sas       20   0 4574020 1.725g   8820 S   0.3  1.8 144:01.34 java  6470 sas       20   0 2820028 1.019g   8352 S   0.0  1.1  63:41.74 java 21250 sas       20   0 6095888 0.978g   8396 S   0.0  1.0  48:18.93 java 20696 sas       20   0 3696948 992272   9460 S   1.3  1.0 410:26.14 java 22441 sas       20   0 2799880 952356   7772 S   0.0  1.0  51:18.43 java   Using the PID from the top coomand it is possible to track the process back to a service.  4659 - sas-viya-microanalyticservice-default  6470 - sas-viya-modelrepository-default 21250 - sas-viya-parse-execution-provider-default 20696 - sas-viya-report-data-default 22441 - sas-viya-themes-default   One of the actions that are recommended in the SAS note mentioned above is to restart the node where the launcher/compute server are deployed. In this shared environment we wanted to avoid bouncing the machine; therefore it was decided to restart the top five services with the hope that restarting them would free enough memory to be able to launch a compute server.   Here are the results captured immediately before and after the restarts.   Service Before (GB) After (GB) sas-viya-microanalyticservice-default 1.73 .33 sas-viya-modelrepository-default 1.02 .91 sas-viya-parse-execution-provider-default 1.00 .34 sas-viya-report-data-default 1.00 .45 sas-viya-themes-default .90 .40   Now if we check memory at the system level we see that 2 GB are now free, compared to 0 GB before, and a total of 5 GB is available for starting new applications.     Once the five services were restarted an attempt to logon to SAS Studio 5 was successful and the launcher was able to launch a compute server.   But what is the memory limit that triggers this situation? Some Additional Testing Although the aforementioned SAS note provides some diagnostic steps that indicate lack of memory may trigger this scenario, no documentation could be found on configuration settings that manage the value that triggers it. The SAS Viya Administrator's documentation mentions memory warnings that are triggered when memory utilization hits 95%, but it makes no mention of the impact when it exceeds that level. So a little trial and error testing was in order.   A program that progressively consumes memory was needed in order to test the threshold. An internet search quickly provided a simple solution. The following C program once compiled and executed will consume 1 GB of RAM every 20 seconds. The interval of 20 seconds was chosen to allow a logon attempt for each gigabyte of RAM consumed.   #include <malloc.h> #include <unistd.h> #include <memory.h> #define MB 1024 * 1024 int main() { while (1) { void *p = malloc( 1000*MB ); memset(p,0, 1000*MB ); sleep(20); } } Testing with Viya Services Split across Two Machines Testing was initially performed on a different collection of SAS VMs that were used for another workshop. This collection spread the SAS Viya services over two machines, each with 32 GB of RAM. In this collection the intviya02 machine hosted the services in question.   The above program was started and the dstat command was used to track memory usage every 10 seconds. (Unfortunately, dstat doesn't calculate and report the available memory.) Once the dstat command and the memory program were started a logon to SAS Studio 5 was attempted every 20 seconds.     When free+cache memory on the machine that was hosting the launcher and compute server fell to approximately 1.5 GB, the launcher failed. On this machine, that level represents approximately 5% of total memory. Testing with SAS Viya Services on One Machine What about the shared environment where all Viya services were on one machine? Testing was completed there as well. This machine has 94 GB of RAM.   Here you can see from the dstat output that memory is steadily consumed until free memory falls to 524 MB.     When the free+cache memory on this machine fell to a little over 7 GB of RAM, the launcher server was not able to launch a compute server. This is approximately 7% of memory. But keep in mind that typically not all of the cache can be reclaimed. So in this case, where over 6GB of memory is used by the file cache, it is likely that the available memory is closer to 5%.   We conclude from these tests that when the 95% threshold is exceeded, the launcher server will no longer launch compute servers. It should be noted that Linux would be able to handle exceeding this value, but it might incur paging, which would result in performance degradation. How to Manage this Scenario? Although this scenario involved SAS Studio 5, it is expected that the launcher server will fail to launch compute servers for other applications. So what action can you take to provide both an immediate and a long-term fix? Short Term Although it is not ideal, you can take a similar action to the one taken above. Identify large consumers of memory and restart them to temporarily free memory. Of course this may not be an option in a production environment. Users won't be too happy if you pull the rug from underneath them. However, depending on application usage you may not have a choice. For example if the users are primarily using Model Studio and Visual Forecasting, both clients of the compute server, it may be necessary. Keep in mind that memory usage of restarted services will likely continue to grow and the environment may eventually return to the same state.   In the shared environment mentioned above, where the initial failure occurred, it was discovered that Anaconda's JupyterHub was not terminating user sessions. In addition to restarting SAS Viya services, many of the old unused Python sessions were terminated to free additional memory. If there are other non-SAS applications deployed on the same machine, terminating these may be another option to reduce and manage memory usage. Long Term Of course the long term solution is to add memory. With additional memory, there will be room for growth in the future. Adding memory is typically more challenging on physical machines than on virtual machines. Future Deployments If you are planning a SAS Viya deployment in the near future, ensure that the machine hosting the launcher and compute servers has adequate memory to accommodate some growth. The SAS Viya services in the shared SAS test environment consumed approximately 68 GB when services were first started. System memory usage grew to 85 GB over several months. But keep in mind that 85 GB includes the idle JupyterHub sessions that were never terminated. The "after" snapshot of the free command shown earlier does not include the memory that was freed by terminating these sessions. You can be sure memory consumption will vary depending on the products deployed and type of work being performed.   To assist with memory estimation, you can view this blog, which captured baseline memory usage by host group and service for a pre-production Viya 3.4 deployment (SAS Ship Event 18w30 - shipped in June 2018). We have since posted an updated blog on memory usage from a 19w21 deployment (shipped in May 2019). A Brief Consul Perspective The SAS Viya service registry component, SAS Configuration Server, relies on HashiCorp Consul.  While this testing was in progress the Consul web interface provided additional information. For example, if you use the Consul web interface to review the status of services once the memory threshold has been exceeded you will notice a large number of services with "Failed" status. Here is a partial view of the failing services:       This status report is somewhat misleading because all of these services didn't truly fail. If we look at the launcher server, Consul reports that the Memory percent free (memory-check) is showing a Warning status. The failures indicate that each of thses services failed to pass a check, which in this case is a memory check.       And drilling into this warning shows that if memory utilization exceeds 95%, the warning is triggered. This confirms the requirement for 5% of free memory that we saw earlier. What Consul can't tell you is that this status will prevent the launching of a compute server.     So even though the Consul status of services provides a warning when the memory threshold is breached, it is not clear from a SAS Viya perspective what the implications are.   Ideally it would be worthwhile to set up a system alert notifying an admin when Used Memory approaches 95%. This would significantly reduce the possibility of encountering a situation where the launcher will not fire up a compute server. Final Thoughts The launcher server performs a memory check prior to launching a compute server. If 95% or more of the memory is in use at that moment, the launcher server will fail to launch a compute server. Although extensive testing has not been undertaken to see if other service failures occur, it is likely that the launcher will also fail for other clients.   For existing deployments the immediate options for environments that encounter this scenario are limited to trying to free memory by terminating or restarting existing processes. The longer-term option is to add memory, but this action will likely require an outage. With new deployments, you should therefore ensure that the environment is configured with adequate memory to start. The Deployment Guide that accompanies SAS Viya software orders provides memory guidelines in the System Requirements section.   Thanks for reading. And if you have additional information that you think others would find useful, please share. ... View more

A little more than a year ago we measured the memory footprint of non-CAS host groups and services for a SAS Viya 3.4 order shipped in June 2018 (18w30 ship event, in SAS parlance) in this blog.  One would expect that the memory profile of subsequent ship events would change, but by how much?  In this blog we will look at the shift in memory usage that has occurred in a deployment of a SAS Viya 3.4 order that went to production in May 2019 (19w21 ship event)  when compared to the 18w30 order from a year ago.   Memory is cheap. Why do we care? One of the architect’s tasks is to ensure that each host of SAS Viya services is configured with enough memory to reliably start and run the services with the expected load.  In many environments that are memory-rich, this is not an issue.  But if you read my recent blog about memory requirements, you know that when a machine does not have enough memory,  certain servers might not start.  Therefore memory allocation is one of many important steps of the planning for a SAS Viya 3.4 deployment.   One scenario we’ve seen where this can become a factor is when a customer has a limited selection of memory sizes for their virtual machines used for deployment.  For example, when only virtual machines configured with 32GB of memory are available for SAS Viya services, the services must be spread across multiple machines.  The mechanism for assigning services to machines is called host groups. In this scenario it is important to understand the host group memory profile and to distribute the host groups in a manner that ensures each machine has enough memory for the short and long term.   Most of the key points regarding host groups and topology were highlighted in my blog from a year ago, so we’ll get right into the comparison.   The Comparison To recap the testing that we discussed in the previous blog, we wanted to determine whether SAS R&D had delivered on a promise to reduce the memory usage of SAS Viya 3.4 (July 2018), after memory requirements had increased in SAS Viya 3.3. We developed a method of finding the service associated with each SAS Viya component. We then mapped each service to its host group, which corresponds to a file name in the group_vars directory in the playbook. Finally, we used the PID for each service to check its memory usage.   Now we want to know whether the May 2019 release of SAS Viya 3.4 resulted in a further reduction in the memory footprint of its services.   Orders The SAS Viya software orders that we used for comparison were orders that we used for deployment in our SAS workshop environment. Here are the details so that SAS employees can reproduce our tests (the ordering system used for testing is not available outside of the SAS network):   The preproduction 18w30 order from last year included the following products: The productionn order for 19w21 added more products: SAS Intelligent Decisioning, SAS Analytics for IoT and SAS Risk Engine.    The additional products mean that the comparison is not like-for-like, but the differences don’t make our reporting any less useful because the additional products brought along their own host groups.   An item of note is that while 18w30 was a fresh deployment, the 19w21 environment was not.  It was based on the original 18w30 deployment which was subsequently updated to production, updated again at 18w39, and finally shipped again at 19w21.  This should not impact the reporting.  And in this environment all non-CAS services run on a single host with 96GB of memory.   Host Groups If we look at the difference in host groups using the diff command, we see the following new host groups in the 19w21 order.  (The diff command was used to discern the differences, but only the relevant host group information is displayed.)  It is not surprising that there is one host group per added product.   # The AIoTServices host group contains the services for SAS Analytics for IoT. [AIoTServices] # Deploys the SAS High-Performance Risk analytics cluster installation script. [HighPerformanceRiskGridInstaller] # The subjectcontacts host group contains a service of SAS Intelligent Decisioning that records contacts and responses of treatments. [subjectcontacts]   Total memory used for all host groups Now that we know the difference in host groups, let’s look at a summary of overall memory usage. If you recall from the blog last year, the script walks through each service of each host group by scanning files in the group_vars directory of the SAS Viya playbook.  It captures the rss (resident set size) and vm (virtual memory) size of each process tracked to a service.  Some of the services are manually mapped to a host group as the mapping is not in the playbook.  The script was executed not long after the services were all running and before any activity of significance was initiated.  Therefore these measurements are a baseline.   Remember that these numbers change constantly, even on an idle system.  An idle Viya system may be idle from a user perspective, but there is always some activity within the system.   In the following table we see the summary of memory used, real and virtual, as well as the total number of services when the two ship events are compared.   Notice that the amount of total memory used is basically the same even though three products were added and the deployment consists of eleven additional services.  Based on this mix of products it appears that the memory footprint is roughly the same, but we can still assume that it shrank, given that the number of products and services had increased.   The total memory used is approximately 67GB (68,858 / 1024) but that value includes some double-counting of shared memory.  For example, the pgpool processes, of which there are over one thousand, access shared memory, but the script that calculates the numbers is not aware of shared memory.  Therefore the actual amount of required memory is slightly less.  A little more on this topic later.   More details If you read the blog from last year you recall that this information was placed in a spreadsheet to summarize information from the service/process level up to the host group level and then the overall level.  The same process was used this time;  new tabs were added for 19w21 and then a comparison was made between the two ship events.   The following screen shot from the spreadsheet comparing data from the SAS Viya software from the two ship events. The last three columns calculate the differences among the three metrics.  The differences of significance have been highlighted using conditional highlighting, showing decreases or increases in memory.  The highlighting is a little counterintuitive because negative is “good” (memory footprint shrank) and positive is “bad” (memory footprint grew).   A quick review of the chart shows that the host groups with the largest decrease in memory (>300MB) used are CoreServices, espServer, Model Services, pgpoolc, ReportViewerServices and VisualTextAnalytics.  The host group with the largest decrease, espServer, consisted only of the esmagent at 18w30.  The SAS Event Stream Manager agent is no longer shipped at 19w21.  It should also be noted that the ESP server, although not a service, has the potential to consume a fair amount of memory.    However, in most deployments that include SAS Event Stream Processing, the ESP server will be deployed on a dedicated machine(s).   The host groups that increased the most (>300MB) were Advanced Analytics, DataMining, DataServices, ScoringServices and VisualForecasting.  Notice that most of the host groups that increased in memory usage also increased in the number of services.  Two of last three obviously show as an increase because they are new, AIoTServices and subjectcontacts. And although it is not new, espStreamviewer was not defined as a service at 18w30.   All things considered, the memory footprint has decreased between the two ship events.  The difference is noticeable but not too significant.   What about growth? An idle system is a good place to start, but it doesn’t provide insight into the services that will grow with user activity.  The shared VLE environment used for SAS training workshops is a good environment to capture this information.  Students regularly enroll in classes and connect to the environment.  Once logged on, they run a wide variety of tasks, both in the web interfaces and via open source interfaces such as Python via JupyterHub.  Although it doesn’t simulate a customer scenario, the VLE environment provides a wide variety of activity that drives a modest amount of resource usage.   Data was captured from this environment after eight days of usage.  Over this period approximately 40+ users logged on.  That data was then compared to the baseline shown above.  The following screen shot shows the comparison of an idle versus active shared SAS testing environment.   As we would expect many of the host groups experienced memory growth.  The Grand Total row shows that the total memory used grew nearly 10GB (~ 15%).   The host groups experiencing the greatest growth can be seen in red-shaded cells.  Looks like VisualTextAnalytics is the winner.   Its memory usage grew about 1.6GB.  Upon drilling into that host group, we see that there is one service, sas-viya-parse-execution-provider-default, that accounts for a significant portion (~800MB) of that amount.   Several host groups had decreased memory usage.  These changes are most like the result of stolen idle pages that are no longer in memory.   Although this data can provide some insight into the host groups whose memory requirements grew the most in a multi-user environment, chances are that it is not entirely representative of a realistic (customer) site.   Even more detail… I could highlight additional details here, but I think you get the gist of the changes.  Now is when the exploring begins.  Open the spreadsheet via the link at the end of this blog.  It contains nine tabs: HG_Memory_18w30– raw data from the script (idle) HG_Memory_19w21– raw data from the script (idle) HG_Memory_19w21_Active– raw data from the script (active) HG_Memory_Pivot_18w30– pivot table from data (idle) HG_Memory_Pivot_19w21– pivot table from data (idle) HG_Memory_Pivot_19w21_Active– pivot table from data (active) CompareHG– host group comparison (screen shot above) CompareHG_Active– host group comparison of 19w21 idle versus active CompareHGServices– host group and service comparison of raw data (18w30 vs 19w21) Within the pivot tables you can expand the host groups and view the services associated with each host group.  The last tab, CompareHGServices, is an easy way to view the differences at the services level.  This data in the various tabs can be used to group and assign host groups to multiple hosts in a multi-machine deployment of SAS Viya.   A useful tool While revisiting this topic and looking at shared versus private memory, I came across a tool that I believe you may find useful if you have an interest in digging into process-level memory detail. The tool, ps_mem, can be installed via yum or pip on Red Hat Enterprise Linux or CentOS.  It is a Python package that provides numerous ways to summarize private and shared memory usage by process, user, etc.   https://github.com/pixelb/ps_mem First, here is the command help:   ps_mem -h Usage: ps_mem [OPTION]... Show program core memory usage -h, -help Show this help -p <pid>[,pid2,...pidN] Only show memory usage PIDs in the specified list -s, --split-args Show and separate by, all command line arguments -t, --total Show only the total value -d, --discriminate-by-pid Show by process rather than by program -S, --swap Show swap information -w <N> Measure and show process memory every N seconds   The first example will show the private and shared memory of every process, sorted by ascending total.  The output has been truncated as it is quite lengthy.   ps_mem -s Private + Shared = RAM used Program 96.0 KiB + 8.5 KiB = 104.5 KiB erl_child_setup 150000 124.0 KiB + 13.5 KiB = 137.5 KiB /sbin/agetty --noclear tty1 linux 160.0 KiB + 14.5 KiB = 174.5 KiB /usr/bin/lsmd -d 168.0 KiB + 25.5 KiB = 193.5 KiB /usr/bin/rhsmcertd 168.0 KiB + 27.0 KiB = 195.0 KiB rhnsd 192.0 KiB + 11.5 KiB = 203.5 KiB /sbin/rngd -f 204.0 KiB + 14.5 KiB = 218.5 KiB /opt/sas/viya/home/lib64/erlang/erts-9.2/bin/epmd -daemon 228.0 KiB + 32.0 KiB = 260.0 KiB /usr/sbin/atd -f 236.0 KiB + 25.0 KiB = 261.0 KiB sasels 34 31 35 2 f9dbe 232.0 KiB + 30.5 KiB = 262.5 KiB sasels 7 4 8 2 7efd9 . . . .   Now let’s show memory usage for the RabbitMQ pid:   ps_mem -p 19216 Private + Shared = RAM used Program 586.1 MiB + 410.0 KiB = 586.5 MiB beam.smp --------------------------------- 586.5 MiB     Here is memory usage for the sas user account by process name, sorted by RAM used.  The number following the process name indicates the number of processes.   ps_mem -p $(pgrep -d, -u sas) Private + Shared = RAM used Program 6.6 MiB + 2.1 MiB = 8.6 MiB objspawn 6.8 MiB + 2.3 MiB = 9.1 MiB cntspawn 8.1 MiB + 1.3 MiB = 9.4 MiB launcher 10.3 MiB + 47.0 KiB = 10.4 MiB sas-run-launcher 17.0 MiB + 54.5 KiB = 17.1 MiB sas-mbagent 23.1 MiB + 57.5 KiB = 23.1 MiB sas-alert-track 33.4 MiB + 78.5 KiB = 33.5 MiB sas-watch 35.8 MiB + 7.0 KiB = 35.8 MiB vault 40.6 MiB + 76.5 KiB = 40.6 MiB sas-stream 66.2 MiB + 4.2 MiB = 70.4 MiB consul-template (5) 66.9 MiB + 6.1 MiB = 73.0 MiB sas-ops-agent (2) 152.5 MiB + 8.5 KiB = 152.5 MiB consul 514.1 MiB + 158.0 MiB = 672.0 MiB postgres (220) 1.3 GiB + 111.0 MiB = 1.4 GiB pgpool (1027) 64.3 GiB + 43.0 MiB = 64.3 GiB java (159) --------------------------------- 66.9 GiB   Finally, the output shows totals by all users on this host.   for i in $(ps -e -o user= | sort | uniq); do printf '%-20s%10s\n' $i $(sudo ps_mem --total -p $(pgrep -d, -u $i)); done apache 41384960 chrony 757760 dbus 1084928 ldap 61241856 libstoragemgmt 220672 polkitd 8229376 postfix 2945024 root 1115027968 rpc 771072 rpcuser 1070592 sas 71785367552 sasrabbitmq 614343168   As you can see, most of the memory is consumed by sas and sasrabbitmq.  There are a few processes that run as root, but those have a very small memory footprint.   When the Python tool is run on an active system, we see processes initiated for individual users (the output has been truncated).  Notice that the memory consumption varies from 4 to ~360MB.     for i in $(ps -e -o user= | sort | uniq); do printf '%-20s%10s\n' $i $(sudo ps_mem --total -p $(pgrep -d, -u $i)); done apache 155784704 chrony 335872 dbus 784896 gatedemo033 360659456 gatedemo034 271870976 gatedemo102 283167232 gatedemo106 4552192 gatedemo110 4660736 gatedemo115 4550144 gatedemo121 200544256 gatedemo122 341491200 gatedemo126 4544000 . . . .   If we look at user activity for one of the accounts, gatedemo033, we find that this user initiated a compute server.  Keep this in mind when planning host group placement.  An environment with many SAS Studio users will see at least one session per user.  Other processes were the result of Python sessions initiated by JupyterHub.   ps_mem -p $(pgrep -d, -u gatedemo033) Private + Shared = RAM used Program 240.0 KiB + 65.5 KiB = 305.5 KiB compsrv_start.s 340.4 MiB + 3.3 MiB = 343.7 MiB compsrv --------------------------------- 344.0 MiB =================================   If you have a SAS Viya environment where you want to know the memory footprint, this tool is a quick way to determine memory usage. We recommend using this Python tool before you plan your host group assignments.   Final Thoughts This blog revealed a lot of data captured from three different systems, so let’s try to summarize what we discovered.   A SAS Viya 3.4 deployment at 19w21 used about the same amount of memory as an 18w30 order, despite including three additional products and new services that were added to existing host groups.  Therefore we conclude that the memory footprint of existing services shrank over that period of time.  However, the change (~2-3GB) is likely not significant enough to change the way host groups are distributed across multiple machines.   There is about a 15% increase in memory usage when an active 19w21 environment is compared to an idle system.  Twelve of the 37 host groups account for most of this growth.  Remember to keep growth in mind when planning to distribute host groups across a single machine or multiple machines. YMMV.   If you want to assess the shared and private memory of SAS Viya processes on a deployed system, be sure to check out the ps_mem Python package.   Finally, the conversion of services to the Go language in SAS Viya 3.5 is expected to dramatically change the memory landscape.  We look forward to assessing the memory footprint of this change in the near future. ... View more

One of the anticipated updates for SAS Viya 3.4 is the reduction in the memory footprint.  A deployment of a SAS Viya 3.3 order containing a majority of products uses approximately 75GB of memory before any tuning changes or usage of the system.  This consumption raised some alerts with customers and as a result, SAS R&D has responded with changes that reduce the memory footprint in SAS Viya 3.4.   Since the memory required for a deployment may impact how services are distributed across machines, it would be helpful to capture information at the level of host groups.  Therefore, in this blog we take a first look at the memory consumed by SAS Viya services, broken out by Ansible host group.  To be clear, this blog is not about comparing memory usage of SAS Viya 3.3 to 3.4, but more about the breakdown of memory usage in a SAS Viya 3.4 deployment.   The Topology Impact As you might imagine, the memory footprint of a SAS Viya deployment has implications when sizing machines and determining the topology.  The GEL team regularly creates collections of test machines with a limited number of machine sizes, primarily with 16 and 32GB memory configurations. Therefore, it is important to understand the estimated memory footprint of services so that memory is not over-committed on any of the machines of a deployment.  If one machine is overcommitted, it is likely that performance will suffer for the applications on that machine as a result of paging and swapping.  So ideally we want to make the most efficient placement of services on each machine in order to minimize the number of machines we use.   This phenomenon is not limited to SAS’ testing environment. Customers who deploy on virtual or physical environments where there is a limited memory on each configured machine may need to divide the SAS Viya services to effectively use memory.  To do this efficiently it will be beneficial to know the memory required for the services that will be placed on each machine.  A customer could use trial and error to assign host groups to machines, but this could end up being very time-consuming.    The question of how to distribute services across machines originally surfaced in the early days of SAS Viya, but prior to SAS Viya 3.3 there was a smaller number of services in a deployment, and thus the memory footprint was smaller.  When Viya 3.3 was released, it came with a significant increase in microservices, and consequently an increase in memory footprint.  Although SAS Viya 3.4 has a modest increase in services, we’ll see shortly that the memory footprint has definitely shrunk.   Host Groups The concept of host groups is likely a familiar topic for SAS Viya customers, but we’ll touch briefly on them here.   The unit of configuration used to manage placement of SAS Viya software components and services is an Ansible host group.  Ansible host groups are defined within the Ansible inventory file.  Host groups are labels that represent one or more software components and services to be deployed to specific machine(s).  The number of host groups within the inventory file is variable and determined by the products in the order.  Host groups within the inventory file are generated during playbook creation.  Each service within a host group will have unique memory requirements. Therefore, ascertaining the memory required for the service(s) within a host group will help with placing the services on machines so that memory is not over-committed.   A host group within the inventory file is specified in brackets.  For example, the following snippet from the inventory.ini file identifies the AdvancedAnalytics host group.  It is followed by a nickname, deployTarget, which relates to a specific host machine where the components of the AdvancedAnalytics host group will be installed.  Each host group in the inventory file is followed by one or more unique nicknames, which correspond to unique machines.   [AdvancedAnalytics] deployTarget   If you need more information about host groups, they are described in the SAS Viya 3.4 Deployment Guide.   Determining Memory Usage per Host Group The key to capturing the amount of memory used by host group is attaining the mapping of services to host groups.  If this mapping can be generated, then it increases the chances of capturing the metrics to calculate host group memory consumption.   Erwan Granger, a SAS colleague, made an attempt back in November 2017 to capture this information by creating a pre-production SAS Viya 3.3 order with all products and then deploying each host group on a dedicated, very small machine. He created a testbed in AWS with 37 machines.  The purpose was two-fold: To see if it would actually deploy one host group per machine. To capture the amount of memory used by each host group.  His deployment was successful, and he was able to capture memory usage at the system level.  And although it was a great idea to capture memory requirements without requiring a host group mapping, it obviously is not very useful.   After some recent digging, it was determined that one section of the generated playbook is the key to mapping most of the services to host groups.  The group_vars directory within the playbook contains files that map to the host group names. Shown here is an abbreviated list of directories that match host groups in the inventory file.   total 348 -r--r--r--. 1 sas sas 3472 May 25 15:12 AdminServices -r--r--r--. 1 sas sas 2884 May 25 15:12 AdvancedAnalytics -r--r--r--. 1 sas sas 4954 May 25 15:12 all -r--r--r--. 1 sas sas 2400 May 25 15:12 CASServices -r--r--r--. 1 sas sas 2866 May 25 15:12 CognitiveComputingServices -r--r--r--. 1 sas sas 5597 May 25 15:12 CommandLine . . .     Within each of these files are the software components associated with that host group.  For example, device-management is a software component within AdminServices.   device-management: SERVICE_APP_NAME: deviceManagement SERVICE_MEMORY_NEEDED: 512 SERVICE_PRODUCT_NAME: device-management SERVICE_YUM_PACKAGE: sas-device-management   After some testing, it was determined that the value of SERVICE_PRODUCT_NAME could be used to resolve the service associated with that component.  And as a result, it is possible to map the service to the host group, which is the file name in the group_vars directory. A search in /etc/init.d for the string “device” reveals the following:   [root@intviya01 init.d]# ll *device* -rwxr-xr-x. 1 sas sas 6581 May 17 20:56 sas-viya-device-management-default   Notice that the value of the field for SERVICE_PRODUCT_NAME identified earlier matches the third and fourth qualifier of the service name.  And it turns out that this holds true for most services. Based on this discovery, a script was written to scrape the value for this field, generate the service name, and associate the service name to the host group.   The files in sas_viya_playbook/group_vars contain many components, not all of which have an associated service.  So it is necessary to use the SERVICE_PRODUCT_NAME value to resolve the service name and check to see if it exists in /etc/init.d.   The shell script mentioned earlier was written to step through each host group file, resolving a service name and checking to see if it is defined in /etc/init.d.  Using a SAS software order that contained all but a few of the SAS Viya 3.4 products, this method resolved 155 out of 172 services.  The remaining services were captured manually and appended to the generated list.   At this point, we have a mapping but still no memory metrics.  There was still some work to do to capture memory usage.   How can we associate SAS Viya services with running processes?  It turns out that it is easier than you may think.  When a service is started, it captures the PID (process ID) in a file and stores it in /var/run/sas under the service name with a final qualifier of “.pid”.  Shown here is the service identified earlier:   [root@intviya01 sas]# pwd /var/run/sas [root@intviya01 sas]# ll *device* -rw-r--r--. 1 sas sas 6 May 31 00:27 sas-viya-device-management-default.pid [root@intviya01 sas]# cat sas-viya-device-management-default.pid 25458   Now that we have the PID for the service, we can use the “ps” command to check the memory usage.  The following command returns the memory used (the RSS, or resident set size) in kilobytes.  In this case, the memory used was approximately 452MB.   [root@intviya01 sas]# PID=25458;ps -p $PID -o rss= 452524   Finally, to generate the big picture of memory usage, this data is written to a file.  The fields that are written are host group, service name, resident set size, and virtual memory allocated.   Once the file is created, it can be dropped into Excel and a Pivot table created (I know, I know. . .). Before we look at the details of the spreadsheet, let’s look at the system-level view of memory using the free command:   [root@intviya01 tmp]# free -m total used free shared buff/cache available Mem: 96520 67456 4428 1237 24636 26810 Swap: 8063 0 8063   It appears that approximately 67GB out of 96GB of memory are used. Since we now know how much memory is in use from a system perspective, let’s look at the memory used by host group.  Here is a snippet of the summary by host group:                                                  . . . .     If you noticed that the sum of the memory used as reported by the ps command is greater than the system level memory used, I believe this is because the amount of memory used reported by the ps command includes shared memory.  The free command indicated that there were 1237MB of shared memory.  So if that amount of memory is removed from the total shown by host group, then the totals are fairly close.  And keep in mind that we are looking for ballpark numbers for planning.  These values will change (in most cases, they will increase) once users are active on the system.  (See additional caveats below.)   …and here is a snippet showing the first two host groups expanded:     From this view you can clearly see each service and how much memory it uses.  The full spreadsheet is available below.   Caveats There are several caveats that need to be mentioned: Memory usage for RabbitMQ and pgpool were captured in a manner different that the rest of the services. There are over 1000 pgpool processes created when SAS Viya starts. These are not captured as part of the services’ memory profile.  Values for these processes were captured based on the process name and then summed.  The pgpool processes are driven by the connection pool settings for the SAS Infrastructure Data Server. Because there is no pid file for RabbitMQ, all processes running under the rabbitmq userid were captured. As discussed earlier, not all services mapped properly, requiring manual assignment. The file containing data for those services is available below. There is no simple way to determine if all PIDs related to the deployment were captured. This script currently only works on a single-machine deployment. However, it could be modified to work across machines (which requires time). The script is quick-and-dirty code used to capture basic data and could use refinement. But I believe it provides enough value to be used for planning a deployment of services across machines at this point.   Quick Note about SAS Viya 3.3 vs SAS Viya 3.4 One of the SAS Viya 3.3 test environments used by GEL for workshops has a similar deployment of products to the environment that was used for SAS Viya 3.4 testing. SAS Viya 3.3 used approximately 74GB of memory SAS Viya 3.4 used approximately 65GB of memory for a newly-started idle system; however, this included over 3GB used for SAS Event Stream Processing services, which were not in the SAS Viya 3.3 order. It also included new host groups, such as StudioViya). So there was definitely a noticeable drop even though we aren’t comparing “apples with apples” and new functionality has been added.  To gain more in-depth insight requires more research and time spent comparing like orders.   Final Thoughts Deploying SAS Viya 3.4 may require spreading the services across more than one machine. Ensuring that the services are efficiently placed across machines requires understanding the memory profile for each host group.  The script that I discussed in this blog is an initial attempt to capture data for that profile so that informed decisions can be made during the process of planning the deployment.  The same script could be rerun to capture memory usage after numerous users are logged in and using the system.  The delta could then be calculated to gain insight into which services are prone to memory growth.   See Excel spreadsheet attachment for a memory breakdown of host group/services.  ... View more

The integration of SAS 9 and Viya has steadily evolved over the past couple of years and has become quite robust. Several articles have been written to show the configuration required to access CAS from SAS 9.4 clients using SAS/CONNECT as well as the using SAS 9.4 as a direct CAS client. However, questions regarding SAS 9.4 clients communicating with SAS Viya and CAS services continue to surface. As a result it seemed like a good idea to briefly revisit this topic. Intro Many new SAS Viya customers are existing SAS 9 customers. They are familiar with the SAS 9 client interfaces and want to understand how to configure the environments to talk to each other. This can be confusing at times as some of the component names are the same and others are different. In addition the requirements change depending on the levels of software. As a result these topics raise questions.   Here are some of the questions that have been asked. Do I need to use SAS/CONNECT to communicate with SAS Viya and CAS from SAS 9? How do I know if I have built-in integration in SAS 9? Depending on the platform, where do I load the certificates for encrypted communication with CAS? How do I get around a permissions issue when CAS sessions started via SAS/CONNECT run under the SAS/CONNECT account but the CAS path directory is owned by the cas account? Before we dig into the questions, it is advised that you review the following articles as they will provide a partial picture of configuration required to access CAS from various clients.   Configuring Enterprise Miner to chat with CAS - As you may have guessed, this article guides the administrator through the steps to enable EM to talk to CAS   Upload Data to CAS using EG Tasks - This article explains how SAS/CONNECT may be used when using the "Upload to CAS" task when the SAS 9 environment is at 9.4M5 and the Viya environment doesn't include SAS/CONNECT.   CAS and SAS Data Integration Studio 4.903 (on SAS 9.4M5) - This article walks the user through the steps of configuring a SAS 9.4M5 environment to enable SAS DI Studio to communicate with CAS. It also references a prior post that walks the user through configuration to use SAS/CONNECT with SAS DI Studio transforms.   Some of the information in those articles is repeated here to consolidate and highlight the key points.   It may be helpful to have the major related ship events handy.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Do we need SAS/CONNECT? As you've probably figured out, "it depends".   If your customer's SAS 9.4 environment is at SAS 9.4M5 or later and they want to interact with a SAS Viya 3.3 or later deployment then it is not necessary to have SAS/CONNECT in either environment. The built-in CAS integration capabilities of SAS 9.4M5 enable the user to communicate directly from a CAS client.   If your customer's SAS 9.4 environment precedes SAS 9.4M5 and they want to communicate with a SAS Viya environment, then SAS/CONNECT will need to be available in each both environments if they intend to upload, download and analyze data within CAS from a SAS 9 client.   Therefore it stands to reason that for customers at SAS 9.4M4 or earlier, it would be in their best interest to update to SAS 9.4M6. Although SAS/CONNECT has stood the test of time, the built-in integration capabilities of CAS-enabled versions of SAS 9 far exceed the added complication of configuring it to communicate with Viya. Desktop Clients and SAS/CONNECT Some of the workstation clients contains tasks that require SAS/CONNECT. For example the "Upload to CAS" task in SAS Enterprise Guide uses SAS/CONNECT in the generated code to perform the upload. So if SAS/CONNECT is not available in either environment then this task has limited value.   If SAS/CONNECT is available in the SAS 9.4M5 or later environment and not available in SAS Viya, then it is possible to initiate SAS/CONNECT sessions locally which in turn use the built-in integration capabilities to communicate with CAS. This may be of value if you want to run multiple concurrent SAS 9 CONNECT sessions asynchronously where each sessions is performing different types of work or working on unique segments of data. See the "Upload Data to CAS using EG Tasks" article listed earlier. How do I know if I have built-in integration in SAS 9? As alluded to in the last section, if the SAS 9 environment is as SAS 9.4M5 or later then the integration capabilities are available. However, if the deployment is from the initial release of 9.4M5 at 19w38, then fewer integration capabilities are available.   The SAS 9 maintenance level can be viewed at the beginning of a SAS log. In the following example, we see the level is "9.4 (TS1M5)". But that doesn't provide any further information. Please note that the line following the level is the site name, which in this case reflects comments added for this internal order. It turns out this order added products to the deployment at 18w21.     To gain more detail on the release you can use the SYSVLONG4 automatic macro variable to view the software release and maintenance level. In the example below the software date is Sep 13, 2017. This is the week before 9.4M5 was released at 17w38. Therefore this SAS Foundation is from the 17w38 release and would not contain all of the functionality available at 17w47.   %put &sysvlong4 ; 9.04.01M5P09132017   The following is from a SAS 9.4M6 deployment. This ship event would contain all of the most current integration updates.   %put &sysvlong4 ; 9.04.01M6P11072018   If you are familiar with the deployment registry you would find that the macro date matches the "last_port_date" in the deployment registry.     This information combined with the ship event tables shown earlier should help you identify whether it is helpful to upgrade or not.   To better understand the integration capabilities at different release and maintenance levels, see the official What's New doc. Do I need the certificates from the Viya/CAS environment? Those Viya environments that have enabled encryption in their environment, considered a best practice, require that certificates be copied and installed in the SAS 9.4Mx environment. This is to ensure that communication between the SAS 9.4Mx client and the CAS server is encrypted. The default setting in Viya 3.3 and later versions is to enable CAS encryption. Therefore it will be necessary to copy Viya/CAS certificates to the SAS 9 environment.   An example of identifying and importing certificates in a Windows deployment was covered in the Enterprise Miner article. Where are the certificates stored? If the SAS 9 client environment is on Windows then the certificates are stored in the Windows trusted store as noted above. In this case the Microsoft Management Console is used to import the certificates. Once added you should be able to view them in the MMC.     If the SAS 9 client environment is on a Unix/Linux environment the certificates are stored in the SAS trusted certificate store. The SAS Deployment Manager is used on Unix/Linux environments to add certificates to the SAS store. After they are added the updates should be visible by reviewing the following directory.     See the "Configuring 9.4 Clients to Work with SAS Viya" section of the SAS Viya 3.4 Administration Guide for steps to add certificates to a trusted store. Do I need to take any special action to use built-in integration in addition to adding certificates? Some of the tasks in the desktop clients require metadata configuration. Adding metadata objects in the SAS 9 environment will allow a more seamless integration. CAS servers, CAS libraries and authentication domains are some of the key metadata objects that are typically defined. The details are beyond the scope of this post and some examples can be found in the articles mentioned earlier.   In some cases it may be necessary to define a .authinfo or _authinfo file with credentials in order to authenticate to CAS. For example, if the credentials are not added to an authentication domain then an authinfo file can be used for authentication. A possible scenario to consider One question that surfaced was related to accounts, permissions and ownership when loading data from a SAS 9 environment to SAS Viya.   The deployments involved a SAS 9 environment at 9.4M4 and SAS Viya 3.3. Since the SAS 9 deployment was 9.4M4 then integrating the two environments required SAS/CONNECT in both.   The scenario required loading data from the SAS 9 environment to a path CASLIB in the SAS Viya environment. The following steps were taken in an attempt to make this happen. The account used in both environments was sastest1 with matching credentials The directory for the CASLIB was created by the sastest1 account on the CAS controller (e.g. /sasdata/remote) Logged on to Environment Manager as sastest1, an administrator, and attempted to create the CASLIB This failed because the CAS session runs as the cas account and it did not have permissions to access the directory Changed ownership of the directory to cas:sas and was able to create the CASLIB Then from the SAS 9 environment a SAS/CONNECT session was initiated to the SAS Viya environment and this session started a CAS session The CONNECT session attempted to upload data to the new CASLIB, but was unable because of permissions The CAS session started by the CONNECT session was started as the user account, sastest1, and not the cas account. Therefore it did not have permission to update the directory specified for the CASLIB One way around this, although not elegant, is to define a secondary group and place both the sastest1 and cas accounts in the group. Then change the group ownership of the directory to the new secondary group and ensure the write bit is enabled. This will allow online sessions from Environment Manager that run as the cas account, and batch sessions to run as sastest1 account.   A better solution is to use Linux file Access Control Lists. This capability allows a system administrator to override the default permissions and ownership of files or directories. The command to manage access control lists is "setfacl".   In this scenario the initial directory was /sasdata/remote and the ownership is cas:sas. Assuming sastest1 is in the sasusers group, it will be necessary to modify the Access Control List to allow the sasusers group to read from and write to /sasdata/remote. First let's look at the permissions from the ls command before changes.   [root@intcas01 sasdata]# pwd /sasdata [root@intcas01 sasdata]# ll total 0 drwxr-xr-x. 2 cas sas 6 Apr 17 15:44 remote   And let's look at the sastest1 account. Note the only group it is a member of is sasusers.   [root@intcas01 sasdata]# id sastest1 uid=2003(sastest1) gid=2003(sasusers) groups=2003(sasusers)   Check the permissions of the directory using the getfacl command. This result is the same as ls command, just in a different format.   [root@intcas01 sasdata]# getfacl /sasdata/remote getfacl: Removing leading '/' from absolute path names # file: sasdata/remote # owner: cas # group: sas user::rwx group::r-x other::r-x   Next let's modify the ACL to allow all sasusers read and write access using the setfacl command.   [root@intcas01 sasdata]# setfacl -m g:sasusers:rwx /sasdata/remote   Finally review the settings again with the getfacl command. Now we see that sasusers have read/write/execute to this directory, even though the sas group is assigned.   [root@intcas01 sasdata]# getfacl /sasdata/remote getfacl: Removing leading '/' from absolute path names # file: sasdata/remote # owner: cas # group: sas user::rwx group::r-x group:sasusers:rwx mask::rwx other::r-x   With the Access Control List updated for this directory the sastest1 account can now update this CASLIB directory. However it will not be able to modify datasets loaded by other users. This method introduces much less overhead than creating a new group and placing cas and sastest1 in it. Final Thoughts The items mentioned in this post identify some of the questions that have come to our attention over the past several months regarding SAS 9 and SAS Viya integration. There are many resources available to answer these questions, but sometimes the answers are not so easy to find.   If you have an experience with SAS 9 and SAS Viya integration that you feel others would find useful, please share in the comments. ... View more

Viya 3.3 introduced improved integration of ESP 5.1 components and products. Those paying attention to SAS products have heard the ESP products names but may not have time to dive into each product, its role and how deployment of ESP relates to Viya. Based on the number of queries received by the GEL team regarding the integration of ESP, it may be helpful to provide a review of products, components, services and service placement. So in this article we will attempt to provide some clarity on these topics, albeit at a fairly high level.   The Products There are four licensed ESP-related products. Here is a brief summary of products and components, as well as the associated 9-byte codes.   SAS Event Stream Processing (EVNTSTM) - This is the core product around which the other products are built. It provides the engine to process and model events in real-time via the ESP engine. There are several components that are included when SAS ESP is ordered. SAS ESP Analytics (ESPANALYT) - provides additional analytic capability beyond that of SAS ESP. Once considered a separate licensable product, it is now included automatically with SAS ESP SAS ESP Studio - a development tool used to build and generate ESP models using a drag and drop web-based user interface SAS ESP Streamviewer - a web-based dashboard tool used to visualize events as they flow through a model SAS ESM Agent - agent used in conjunction with SAS Event Stream Manager SAS Event Stream Processing for Edge (ESPEDGE) - A smaller footprint version of SAS ESP to allow it to run on edge devices such a sensors, routers, etc. It includes the following components SAS ESP Analytics - adds analytic capability to each ESP for Edge instance SAS Event Stream Processing for CAS (ESPCLUSTR) - ESP capability that stands alongside CAS and provides the ability to run ESP models in a cluster and interact with CAS. Two CAS action sets are available for interactions: loadStreams and espCluster. Must be ordered with SAS ESP and SAS VA. SAS Event Stream Manager (ESPMGR) - A tool used to manage ESP models and projects. Provides the operational tool to centralize and automate model deployment. Simplifies management when many engines are being used. SAS ESP is required for SAS ESM orders.   Deployment Options If SAS ESP is ordered as a standalone product it is installed differently than if it is ordered with another Viya solution. When ordered as a standalone product a small script is provided to perform the deployment. When ordered with a Viya solution, including SAS Event Stream Manager, the deployment is integrated with the Ansible playbook. Under the covers both use yum to install RPMs from SAS's repositories. In both scenarios the Software Order Confirmation email contains a zip file, SAS_Viya_deployment_data.zip, that provides the certificates and license information.   SAS ESP - Standalone A standalone order of SAS ESP directs the installer to move the zip file included in the email to the target machine and unzip. In addition to the certificates and license, this file includes a small script (orderinputs.sh) that contains the repositories and yum groups to install.   The contents of orderinputs.sh script:   # Entitled yum repo definitions entitledRepos=””sas-esp-102-x64_redhat_linux_6-yum” “sas-espanalyt-101-x64_redhat_linux_6-yum”” # Function that adds yum groupinstall list function install_yum_groups() { yum groupinstall “SAS Event Stream Processing Analytics” yum groupinstall “SAS Event Stream Manager Agent” yum groupinstall “SAS Event Stream Processing” yum groupinstall “SAS Event Stream Processing Studio” yum groupinstall “SAS Event Stream Processing Streamviewer” yum groupinstall “SAS Text Analytics for English” } licenseFile=SASViyaV0300_09****_Linux_x86-64.txt   The next step is to acquire the deployment script using a link in the email.     The deployment script is then placed in the root directory of the unzipped file from the email.   Once these pieces are in place the deployment script (customized_deployment_script.sh) is executed as an account with sudo rights or with the root account to perform the install.   SAS ESP for Edge Computing - Standalone SAS ESP for Edge Computing is often included in IoT scenarios. From an order perspective SAS ESP Edge is excluded from full product orders, indicating it is incompatible with a full product order. Therefore ESP Edge will be ordered separately from Viya orders.   The email sent from an order of SAS ESP for Edge Computing includes four separate files: the license file, two certificates and a deployment script.     Deployment of the software is accomplished via the ESP_Edge_Script.sh script. The script includes code to install on Red Hat and Debian-based operating systems. All files from the email are moved to the target machine, the script made executable and then executed to perform the install.   With a Viya Product If SAS ESP is ordered with Viya products (e.g. VA 8.2), including SAS Event Stream Manager, the deployment process uses Ansible to deploy the software. Multiple host groups are included in the inventory depending on which products are ordered. The groupings are shown here along with related services.     From the table above one might be tempted to interpret that the service associated with [espServer], sas-viya-esmagent-default, is used to start/stop ESP servers and load models. This is true only if SAS Event Stream Manager is included with the order. The esmagent service is included with all SAS ESP orders, but only used when SAS ESM is available and used to manage projects, models and ESP servers.   Note there is no associated service for [espStreamviewer]. SAS ESP Streamviewer is started via a script that is installed at deployment time. The script can be found at   /opt/sas/viya/home/SASEventStreamProcessingEngine/5.1.0/bin/streamviewer.sh   SAS ESM contains the core Viya components and services that are included with other Viya products. Included with SAS ESM are consul, pgpool, httpproxy, rabbitmq, sasdatasvrc and CoreServices. In a scenario where SAS ESP and ESM are ordered together, then all of the Viya pieces are available to enable SAS ESM and ESP Studio to authenticate users to LDAP via included Viya services.   Associated Data Stores SAS ESP is all about processing events. However, the developmental and operational aspects of SAS ESP and related products require data stores. SAS ESP Studio, SAS ESM and SAS ESP Streamviewer require a database to store project, model, job and dashboard related objects. When SAS ESP and SAS ESM are deployed together or with other Viya products, objects for ESP Studio and ESM will be stored in the SAS Infrastructure Data Server (i.e. Postgres tables). There is no special configuration required.   Shown below are the schemas and table names for ESM and ESP Studio.     Although ESP Streamviewer is enabled to communicate with and store objects in numerous databases, the data store location shipped with ESP Streamviewer is an H2 database. H2 is a small footprint, file-based database that is included with a standalone ESP deployment. H2 is also the default data store for the ESP Studio in a standalone deployment. If another database is to be used for ESP Streamviewer, it will be necessary to attain the connection and authentication information.   Finally, licensing for ESP is different than most Viya solutions. ESP is licensed by number of events. In order to capture the counts and bits of information needed to properly report usage, it is necessary to start the metering service. The metering service uses an H2 database to save usage metrics. With ESP 5.1, H2 is the only supported database for the metering service.   The following table captures the key components that require data stores and which data stores are defaults.     Host Group Placement In the prior section it was noted that when ESP and related products are deployed with Viya products there are one or more included host groups. You are likely aware a host group is the default unit of control for deploying software per machine. So you might be wandering about topology.   Here are some thoughts regarding host groups and placement.   SAS ESP Server [espServer] Since the ESP Server is the engine that processes events in real-time, it is important that the ESP server not be impacted by other workloads on the same machine. If not managed properly, a resource usage flare-up by non-ESP server workloads could impact the throughput and responsiveness of ESP processing. Therefore it is suggested that the ESP server be placed on a machine by itself when possible.   SAS ESP Studio [espStudio] ESP Studio is a web application that provides an environment for building and testing ESP models. As a result it is suggested that it be placed alongside the other web applications of a deployment.   SAS ESP Streamviewer [espStreamviewer] ESP Streamviewer is a web application that enables the customer to build dashboards fed by events as they flow through ESP models. Like ESP Studio it is recommended that it be placed with other web applications.   SAS Event Stream Manager [viprESM] As you might have guessed by now, SAS ESM is also a web application used for management and operations of ESP models and servers. Therefore, like ESP Studio and ESP Streamviewer, it would feel at home with them along with any other web applications.   One aspect that should be noted about ESP web applications is that there typically won't be a large number of users. Most customers will have only a few developers and testers who design and test ESP models and generate related dashboards. In addition they will likely maintain a small number of operational staff. As a result the load imposed by the web applications will have a much different resource profile than the many of the web applications of other Viya products.   The following diagram shows the ESP components and their placement with respect to other Viya services.     HA Considerations When deployed with Viya some of the host groups can be configured to be highly available. SAS ESP Studio [espStudio] and ESM [viprESM] are capable of being clustered by defining more than one host to the respective host group. When configured in this manner the Apache proxy server is configured to balance requests across the web application instances during deployment. The web applications will use Postgres on the back end to centralize and store database objects.   The same is not true for SAS ESP Streamviewer [espStreamviewer]. Placing multiple machines in this host group will simply ensure that the software is installed. If three machines are defined and all three instances are started there is no coordination across the machines. Each instance of Streamviewer will act independently of each other. In addition it should be noted that they require configuration to use a database. If all instances "point" to the same database then theoretically they should be able to maintain a single copy of each dashboard, rather than three copies. (This is something that I haven't had time to test. Feel free to test and let us know your results. If you start down this road you may as well test updating the proxy settings to balance workload across instances.)   If multiple servers are specified for the [espServer] the deployment process will install the required software for clustering, but clustering ESP servers is substantially more involved than the automatic process for other Viya components. For more information about configuring HA ESP servers, see ESP Failover using Kafka on failover.   I think that's enough for this post. Let's wrap things up.   Final Thoughts SAS ESP can exist on its own or as part of a Viya deployment with other products. The deployment for these two scenarios is quite different. When deploying in a Viya environment it is important to understand the components to ensure they are deployed in the appropriate location. In addition it is also helpful to know with which data store each component is configured. Hopefully this has helped clarify some of the questions about ESP and its relationship to Viya deployments. ... View more

Many SAS 9 customers are familiar with and make heavy use of SAS Enterprise Guide for analytics and reporting. It is a tool on which they rely daily to meet their business needs. As many of those customers transition to SAS Viya there will be a need to access data and capabilities available in CAS from SAS Enterprise Guide. SAS 9 in general, and SAS Enterprise Guide in specific, contain built-in features to assist with this transition. In this post we will walk through the configuration required to assist with uploading data from a SAS 9 environment to CAS.   A couple of months ago Nicolas Robert wrote a very informative article about loading data to CAS from SAS Data Integration Studio. There are obvious similarities with certain portions of the setup and configuration. I recommend reading his post before continuing.   A Query from Halfway around the Globe The GEL team recently received a query from our colleagues in Japan about uploading data from EG and SAS 9.4 to CAS in a Viya environment. The question was specifically if SAS/CONNECT is still required in both the SAS Viya and SAS 9 environments when using the "Upload to CAS" task within EG. My immediate thought was "of course SAS/CONNECT is required in both environments to use this task". But I would soon learn otherwise.   Common Steps with EM and other SAS 9.4M5 Clients In my previous post, we walked through the steps to get Enterprise Miner configured to talk to CAS. If you've read that post or read the SAS Viya 3.3 Administration documentation regarding encryption, you are aware that one of the keys steps required for a SAS 9.4M5 environment to talk with CAS is to store the appropriate certificates from the CAS environment to the SAS 9 environment. The steps for storing these certificates can be viewed in the administration guide and a more general overview can be found in my previous blog. This blog assumes these steps have been completed.   Testing the EG "Upload to CAS" Task (using SAS/CONNECT) In addition to the certificates it will be necessary to verify the SAS/CONNECT setup, create a CAS server and define a CAS libname in metadata. As you would expect these are used as the target for uploading to CAS.   The key steps required to test the connection to CAS are summarized here. Verify SAS/CONNECT setup Define a CAS server definition in metadata Define a CAS libname in metadata Test upload of a local SAS dataset to CAS The "Upload to CAS" task was introduced in SAS Enterprise Guide 7.13 therefore this is the minimum level required. The testing done here used SAS Enterprise Guide 7.15.   SAS/CONNECT If SAS/CONNECT is installed in the SAS 9.4M5 environment and the default configuration is in place then there should be no need to take any action here other than verify the setup. The key item to check is that the Remote Machine defined in the Connect Server - Options tab points to the SAS 9.4M5 machine. This simply means that the CONNECT Spawner will start a CONNECT server on that machine.   In the test RACE environment the deployment was a single Windows machine deployment. Therefore the Remote Host was set to the local host name. In SAS 9.4M4 this would require setting the remote host to the machine where the SAS Viya Connect Spawner was installed. Hopefully this statement will make sense momentarily.                 Click any image to see a larger version.   CAS Server Definition in Metadata Since data will be uploaded to CAS a CAS server definition is required in metadata. This is a simple server definition using the "New Server Wizard" and selecting the "SAS Cloud Analytic Services Server". The key here is to ensure it is pointing to your CAS controller. It is important to verify the "Authentication type" be none. Otherwise it will attempt to add user and password parameters to the CAS library definition, and these are not valid parameters and result in an error.     CAS Library Definition Now that the CAS server has been defined, define a CAS library using the "New Library Wizard". Select the "Cloud Analytic Services Library" template and then the SAS server, specify the CASLIB to be used, define a Libref, and select the CAS server defined in the previous step.         At this point the pieces are in place to test.   Test "Upload to CAS" task You may be a little confused that we didn't define a SAS/CONNECT connection to the Viya 3.3. environment, but just hang in there.   Add a table to the project, such as SASHELP.HUMID, and then add the "Upload to CAS" task to our project. The "Upload to CAS" task will prompt the user with the following message indicating that new capabilities in SAS 9.4M5. What it doesn't mention is that you may have to use the program editor to use these statements. More on this later.     At this point we should have something that looks like this. We need to change the target Library to the library created above and add credentials, as they are not stored.     Our flow looks like this.             If we then run the task, it appears to finish without errors.           First, let's look at the generated code. This may give you a clue why a remote CONNECT session to SAS Viya is not required.     So what happened when it ran? Let's look at the log. It successfully started a SAS/CONNECT session.     In addition it successfully allocated the CAS Public library using the credentials used to start the SAS/CONNECT session.     And successfully uploaded the HUMID dataset to the Public CASLIB. Also note that the file was promoted.     If we check in Environment Manager the table is available.     It is probably pretty clear now why the SAS/CONNECT session to Viya is not required. The local SAS/CONNECT session uses the new SAS 9.4M5 integration capabilities to communicate directly with CAS using the CAS libname defined in metadata. Therefore SAS/CONNECT needs to be deployed in the existing SAS 9.4M5 environment where the EG workspace server resides, but SAS/CONNECT is not required in the SAS Viya environment to load data using the "Upload to CAS" task. Of course data can also be loaded using the native PROCs in SAS 9.4M5 and the program editor within EG. The intent here was to try to use built-in tasks to communicate with CAS.   If you recall earlier it was noted that the SAS 9.4M4 would require defining the SAS/CONNECT spawner and server on SAS Viya. Since SAS 9.4M4 does not contain integration with SAS Viya it is necessary to deploy SAS/CONNECT with SAS Viya so the code from the "Upload to CAS" task can rsubmit code to the Viya SAS/CONNECT server. That code then runs in the Viya environment which has the obvious capability to interact with CAS.   Using Built-in SAS 9.4M5 Integration The "Upload to CAS" task gets the job done using SAS/CONNECT, but ideally it would use one of the built-in integration capabilities of SAS 9.4M5.   One of the potential tasks to perform an upload is the "Query Builder" task. For example the PROC SQL statement generated by the "Query Builder" task reads from SASHELP.SHOES and will attempt to create a table in WORK by default. In order to write to CAS the target library should be changed to a CAS library such as CASPUBLIC.   The following screenshots show key selections required to a load table to CAS.   The first steps are to change the target library to a CAS library and then select "Options for This Query".     By default the query builder will load the table as a session table. In order to promote the table add the "PROMOTE=YES" option to the Options window as seen below.     If you then preview the generated code you will see that the PROMOTE option is added to the output dataset of the query.     With this configuration in place, running the task will load the data to the CAS library and promote it.     From this example you can see that the "Query Builder" task can be used to load data to CAS and promote tables so that they can be used globally. As a result this task is taking advantage of the built-in integration capabilities of SAS 9.4M5.   If the CAS library and related tables are not visible within SAS EG this can be resolved by taking the steps in SAS Note 20760. The following screenshot shows the updates to the library in metadata.     These EG tasks work well when the authentication mechanism is the same for the client and the server, or the credentials are the same in both environments. The credentials used for the client are also used when authenticating to CAS. If this is not the case additional steps will likely need to be taken.   Final Thoughts Although some integration is built into SAS Enterprise Guide not all tasks are updated to take full advantage of SAS 9.4M5 integration capabilities. If you are just loading data to CAS the "Upload to CAS" task will get the job done via local SAS/CONNECT and SAS 9 integration features. The "Query Builder" task may also be used but will require adding the PROMOTE option to the output dataset. A third option is to use the program task to define libraries and load data to CAS.   My thanks to Satoshi Kawakami, Shohei Matsuura, and Nicolas Robert for sharing their thoughts and experiences on this topic.   Feel free to share your tips regarding the use SAS 9.4M5 integration features within SAS Enterprise Guide below.   Thanks for reading! ... View more