In this SAS demo, Raphaël Poumarède (@RPoumarede) of SAS gives a quick introduction to the Google Cloud Platform (GCP) QuickStart for SAS Viya 3.4. He shows how to prepare the Google Cloud environment, configure and execute the Viya Quickstart to run, and monitor and validate an automated deployment of SAS Viya 3.4 in the Google Cloud.     If YouTube is blocked at your site use this link   Raphaël Poumarède (@RPoumarede ) has been at SAS for over 10 years and focuses on SAS architecture and deployment.   Watch more videos like this:  SAS Technical Insights & Expertise Series   Related Resource Google Cloud Platform QuickStart development repository for Viya – https://github.com/sassoftware/quicks... How fast can we get it built? An introduction to the SAS Viya QuickStart for GCP! – https://communities.sas.com/t5/SAS-Co... Google Cloud – https://cloud.google.com/     ... View more

  With the recent updates of the Viya Quickstarts in the Cloud, I thought it would be a good time to come back on the Quickstart process and talk about some recent deployment experience in Azure.   SAS is a "Microsoft One" Commercial Partner which means that SAS and Microsoft teams closely collaborate on cloud technologies, solution areas and rapid deployment models, It makes Azure is a platform of choice for a SAS Viya deployment.   Also, with the move to Containers deployment in Kubernetes for the next Viya 4.0.1 release, we can expect the Cloud platforms such as Azure to become even more popular for Viya deployments.   So if you are an Architect or an Installation Engineer and completely new to Azure, now is probably a good time to get a first exposure to it 😊   The basics of the Viya Quickstart for Azure deployment are covered in the excellent post and 30 minute video from Michael Goddard. Note that the video is also available on YouTube, so any customer can use it to prepare a Quickstart deployment !   The purpose of this blog is to revisit the Quickstart deployment in Azure with a Viya 3.5 order and a CAS MPP environment but also to provide some extra guidance to secure your in Azure and make sure you don’t face some of the issues I encountered in my own deployment 😉   Before diving into the details, I'd like to thank Chris Lynch for his precious help in the write-up of this blog.     What’s new with the Viya Quickstarts SAS's Quickstarts are getting better and better ! Not only have the three of them (AWS, GCP and Azure) have been updated to support the latest Viya version (3.5), but now, they also support CAS MPP configurations.   A new parameter has been added for that in each Quickstart and allows you to have up to 10 CAS workers in your Quickstart deployment. This makes Viya with CAS MPP deployments very easy.   As an example all you have to do is for the Azure Quickstart is to specify the number of CAS workers that you want, in the Quickstart template form (3 for example) :     Each Quickstart documentation (either README files on GitHub pages or PDF documentation for AWS), provides Architecture Diagrams for CAS SMP and CAS MPP cases.   See below the architecture diagram for the Quickstart architecture for SAS Viya on Azure with a CAS MPP configuration.      While the "Quick" word in "Quickstart" means "Quick deployment", it does not mean "Quick (and dirty) Architecture design".   To the contrary, one look a the architecture diagram is enough to see that many important Architecture principles and best practices for a Cloud deployment are implemented by the Quickstarts :   Virtual network for Public/Private subnets separation Gateway/Load-balancer for Web applications access. Bastion Host for the administrative access Network Security Groups to restrict network access to specific ports and source machines Deployment Templates (declarative syntax and automation)   The Quickstarts are not simply provisioning a bunch of VMs to deploy our software. They also provision all the Cloud services or components required to create a robust and secured environment.   Of course, in a customized Cloud deployment, a more sophisticated topology and model could be implemented, but the Quickstart provides a solid starting point, that can be extended as the deployment templates are public.   So, with the addition of MPP CAS, it makes the Quickstart a valid option not only for demo or PoC/V (Proof of Concept/Value) but also for real project deployments for customers licensing a "Quickstart supported" solution (see below).   So let's recap on the changes for our Quick Start !   Viya 3.5 support SAS Visual Analytics 8.5 on Linux SAS Visual Statistics 8.5 on Linux SAS Visual Data Mining and Machine Learning 8.5 on Linux SAS Data Preparation 2.5 Viya-ARK 3.5 integrated Support of CAS MPP (up to 10 workers)   Regarding Azure, please note that there is also a new excellent post from Margaret Crevar (from the EEC team), with considerations on the Azure VM types, network and storage choices that impact the performances of our Software.   Tips and tricks for a Quickstart deployment in Azure    Ok, so now let's zoom on the Azure Quickstart and talk a little bit about the deployment.   As said previously, if you are fairly new to the Viya QuickStart on Azure, the best starting point is the Michael Goddard’s YouTube video : “SAS Viya Quickstart Template for Azure | Introduction and Demonstration”.   If you already know the basics and want to learn more advanced techniques like use Mirror repository or monitor the deployment progress, you will be interested by the following sections.       Use a mirror for your Azure Quickstart deployment For many reasons, using a Mirror repository for the SAS packages is a best practice for any Viya deployment (see this post if if you need to remember why). Using a mirror also allows you to test the Quickstart with an Internal order.   According to the Quickstart README file, to use a Mirror repository, all you have to do is to:   Upload the mirror content into an Azure Blob Storage ("Blob Storage" is the name of the storage service for Azure but each of the major Cloud provider has its own, S3 for AWS or Google Cloud Storage for GCP) with the Azure CLI, And provide the corresponding URI (for the mirror) as a parameter in the QuickStart template (before launching the deployment).   It seem very simple, right ? However, the Azure portal interfaces can sometimes be a little bit confusing, especially if you are new to them. So let's detail a little bit more the steps with a real example.   Upload the mirror   Let's start with the first step: "upoad all the mirror content"     You need a an Azure Storage account. it is recommended to create it in the same region as the Quick Start Deployment resource group. But a good idea is to create the storage account for the mirror in a different resource group from the deployment’s one. Because, then you can remove the deployment by deleting the deployment resource group but maintain the mirror source storage in case you want to redeploy. In a storage account, you can store things in "blob containers", "file shares", "queues" or "tables", here we use the "blob containers". We can create a new container for the mirror called "mirrors" for example.     If you open the Storage explorer in the Azure portal, it should look like this:     Now you need to find the access key corresponding to your Azure Storage account, you should find it in the properties of the storage account ("frarporgdiag" in this example).            You can copy the value of the Key field either for key1 or key2, that's what you will use for your "$STORAGEKEY" in the upload command that we need to run.     Now, you should be in position, to run the Azure CLI command to upload the mirror content into the blob container. The README file of the Quickstart gives the command line syntax :   az storage blob upload-batch --account-name "$STORAGE_ACCOUNT" --account-key "$STORAGEKEY" --destination "$SHARE_NAME" --destination-path "$SUBDIRECTORY_NAME" --source "$(pwd)"     Connect to a machine that has access to the mirror repository source directory (for example located in /mirrordisk/09QSDS) and where the Azure CLI has been installed (for example with YUM) to run the “az” command. Here is a real life  example for the command using the previous example, values:   az storage blob upload-batch --account-name "frarporgdiag" --account-key "kKjO1f4evCuGZ1mC.....your key......UvE+pWsVmQ==" --destination "mirrors" --destination-path "09QSDS" --source "/mirrordisk/09QSDS"   From there, your screen should start to scroll pretty quickly as it reports each file upload into the Azure blob container 😊. As a result, a folder named "09QSDS" is created in your "mirrors" blog container and contains all the files and folder of the mirrored repository.     Actually, if you use the Azure CLI on linux and want to increase the automation of the process, there is even a trick do this all as one line without needing to get the key out of the web interface (by having bash fetch the key for you in a sub var):   az storage blob upload-batch --account-name "$STORAGE_ACCOUNT" --account-key "$(az storage account keys list --resource-group "$RESOURCE_GROUP" --account-name $STORAGE_ACCOUNT --subscription "$SUBSCRIPTION" --query "[0].value" | tr -d '"')" --destination "$SHARE_NAME" --destination-path "$SUBDIRECTORY_NAME" --source "$(pwd)"     Get the mirror URL   So far, so good. Now let's see the second part which is primarily to find the proper URL/URI to use to help our Quickstart deployment to find our brand new Mirror repository 😊   Here is the information that we have from the README file :   Create a SAS key that has (at a minimum) List blob and Get blob privileges on the blob store. During deployment, set the DeploymentMirror parameter to the URL of the folder in the Azure blob that is qualified by that SAS key.   But don’t get confused ! in this context "SAS" means "Shared Access Signature" which is the Azure mechanism to provides secure delegated access to resources in your storage account (such as the "mirrors" blob containers) without compromising the security of your data. See this page for additional details.   Here are the steps from the portal to create the "SAS" key: - In your storage group, select "Shared access signature" on the left side menu. Now create a new one with: - allowed services – blob - allowed resource types- All of them - allowed permissions - Read List   And set the end date far enough in the future to cover your install window.   Then click "generate SAS and Connection String" and grab the value in the "SAS Token" string   If like me, you prefer to use the new “Storage explorer” feature, it would look like this:   Right click on "Mirrors"       Select "Get Shared Access Signature".     Set the expiry time (the "SAS" key used by the Quickstart to download the packages from the mirror repository will only be valid for this period of time), For the permissions, check "Read" and "List". Then click on the "Create" button, to get your URI, that contains the "SAS" key.     We are now in position to build the URL to set in the Viya Quickstart template form ! But before using it, we want to make sure the URL/URI actually works !   So copy the URI that is shown in the portal screen above, for example :   https://frarporgdiag.blob.core.windows.net/mirrors?sp=rl&st=2020-03-20T15:19:05Z&se=2020-03-21T15:19:05Z&sv=2019-02-02&sr=c&sig=WNlrg2w8zO5Ji8jh5KfVvPq379eLIV7OyyoO%2FKgdrcA%3D   If we use this URL directly, it does not work:     We need to manipulate a little bit the url to get the form that is required for the Quickstart template.   Keep the first part of URI string (until the "?"), but then modify the path, as if you wanted to point on a specific file that is available in your mirror repository, for example : entitlement.json.   In our example that would be :   https://frarporgdiag.blob.core.windows.net/mirrors/09QSDS/entitlements.json   Then add the Query string (starting with the "?" that contains the validity dates and the signature/ token)   https://frarporgdiag.blob.core.windows.net/mirrors/09QSDS/entitlements.json?sp=rl&st=2020-03-20T15:19:05Z&se=2020-03-21T15:19:05Z&sv=2019-02-02&sr=c&sig=WNlrg2w8zO5Ji8jh5KfVvPq379eLIV7OyyoO%2FKgdrcA%3D   Now, it is working :     Now, it's time to set the value for the "Deployment Mirror" parameter in the QuickStart template.   So to get the final URL, we just need to "chop off" the entitlements.json, it gives something like that :   https://frarporgdiag.blob.core.windows.net/mirrors/09QSDS/?sp=rl&st=2020-03-20T15:19:05Z&se=2020-03-21T15:19:05Z&sv=2019-02-02&sr=c&sig=WNlrg2w8zO5Ji8jh5KfVvPq379eLIV7OyyoO%2FKgdrcA%3D   It won't work in a web browser, but ,trust me, this is what needs to be set in the Quickstart template.     Monitor and troubleshoot your deployment As you have seen, the value for some of the template parameters can be a little bit tricky to find. So you surely want to monitor your deployment as closely as possible to make sure everything works as expected. Shortly after clicking on the "Purchase" button that launch the Quickstart deployment, the Ansible Controller Virtual machine is created and you can connect to it with the "vmuser" account. As soon as you see in the Azure portal that the "Ansible" Virtual Machine has been created, get the public IP address and connect to it from your ssh client.       You need to provide the SSH private key corresponding that corresponds to the SSH public key that was set in the Quick Start template. From there, you can monitor the deployment log exactly the same way you would to it from an on-premise deployment. For example:   [centos@frarpo-jumphost ~]$ ssh -i ~/.ssh/frarpo-viyadep-key vmuser@52.168.39.103 [vmuser@Ansible]$ tail -f /var/log/sas/install/viya_deployment.log   At the end, you can also check the Viya services status using the Viya ARK MMSU scripts (embedded  in the deployment):   [vmuser@Ansible ~]$ cd /sas/install/ansible/sas_viya_playbook [vmuser@Ansible sas_viya_playbook]$ ansible-playbook viya-ark/playbooks/viya-mmsu/viya-services-status.yml     My deployment has failed…or maybe not? On my first attempt, quite optimistically, I decided to deploy a Viya environment with 3 CAS workers and to keep the default Azure instance types, predefined in the Quickstart template:     But near the end of the deployment I had the unpleasant surprise to see messages below in the Azure Portal for my running deployment :     Humm...It does not look good, does it ? However, using the described technique above I was closely monitoring the deployment via the ansible deployment log and noticed that the deployment log shows no failure for the ansible deployment. Moreover, when I checked the status of my Viya services they all reported an "up and running" status. I even can logon with success to SAS Environment manager where everything seems perfect (with my 3 CAS workers as requested):     So...is my deployment valid or not ?   With a better look at the error message I can isolate an interesting message:   "After restart, services were still found to have not successfully started. This likely indicates that the boxes have been undersized. Try increasing the size of the services box."   Thanks to the Development team, I had an explanation:   "During the install process, the startup of the Viya services becomes very cpu bound. Once everything loads into ram, they settle down a lot, but starting all the services tends to be the point where a larger box is needed for a bit. But the services definitely can come online after the timeouts for the build".   Unfortunately some Azure time out settings (that can not be moved past) introduce a time limit for some tasks that the deployment process cannot meet because of limited CPU resources.   So, although my Viya deployment was successful, my Azure deployment was not.     Of course it would have been better to have a completed successful Azure deployment but as long as there is no failure in the ansible deployment results and all the Viya  services are up and running, we can use Viya as is or increase the instances sizes and redeploy in case we want more reinsurance on how it performs.   Azure deployments are more records of what happened (we can redeploy or undeploy a deployment, but resource groups are really the management object that azure uses).   As  the Quickstart has its own resource group and is expected to be the only thing in that resource group so it can also be cleaned up as a unit (also prevents name collisions).     Conclusion Hopefully after reading this article you know a little bit more about the SAS's Quickstarts (and learned from my own experience and mistakes 😊 ). The Quickstarts are a great way to automate your deployments of Viya in the Public Cloud infrastructure while complying with the acknowledged rules of the art of Cloud deployments. ... View more

In this post, I’d like to discuss some new features introduced with Viya 3.5. I briefly mentioned in a previous SAS Communities article.   This new capability is great as it gives more flexibility to handle complex network configurations at customer sites. However, it requires a careful planning, a good understanding of the concepts and an understanding of the technical requirements/goals.   Rather than paraphrasing the official documentation, the idea for this post is to tell the story of two case studies and the challenges they presented and how they were tackled.   Even though I’ll do my best to explain it as clearly as possible, it is a complex topic. So, if you plan to read the to read it end-end, grab yourself a good cup of coffee or tea and make sure you won’t be disturbed for a little while.   An Example of Multi-NIC Environment (Data Appliance) What is a multi-NIC machine? Why? How do we check that?   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Multi-NIC stands for “Multiple Network Interface Cards”.   Increasingly, servers are coming with multiple network interfaces. The network interfaces could be there to support virtualized environments or in order to have different isolated networks for security, administration and/or performance reasons. Sometimes the term “multi-homed” environment is also used to refer to this configuration.   The network interfaces could be virtual or physical, but they generally come with a different IP address, that is usually associated to a hostname. So, we end up with a single machine having multiple IP addresses and hostnames. How can Viya software be configured to work within this setup?   It is always better to explain a concept using an example, so let’s look at a DA (Data Appliance). In this DA, we have a collection of 5 machines where there are several different networks used for different purposes.   Customers with large/big data, use data appliances to hold data and perform analytics on such platforms. They typically come with a version of Linux, which makes a good fit more for Viya.   In the DA we discuss, there are 3 different types of network involved in the machines network configuration and to consider (Infiniband, 10GbE and 1 GbE). Logging on to the DA we can run a command like ifconfig or ip and check the content of the /etc/hosts file to see that.   bondeth0 Link encap:Ethernet HWaddr 12:6A:0A:25:98:97                inet addr:12.35.250.251 Bcast:12.35.255.255              Mask:255.255.0.0 bondib0   Link encap:Infiniband HWaddr 80:00:05:0A:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00                 inet addr:192.168.10.152          Bcast:192.168.11.255 Mask:255.255.0.0 eth0      Link encap:Ethernet  HWaddr 00:10:E0:B4:89:82                 inet addr:173.16.7.136 Bcast:173.16.255.255              Mask:255.255.0.0 eth8      Link encap:Ethernet  HWaddr 12:69:0A:25:98:97 eth9      Link encap:Ethernet  HWaddr 12:6A:0A:25:98:97 ib0       Link encap:Infiniband HWaddr 80:00:05:0A:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 ib1       Link encap:Infiniband HWaddr 80:00:05:0B:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 lo        Link encap:Local Loopback           inet addr:127.0.0.1   Mask:255.0.0.0 datapp1node01.sci.example.com> Here’s what we see when look at the /etc/hosts file.   127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 173.16.6.136 datapp1node01-adm.sci.example.com datapp1node01-adm 173.16.6.142 datapp1node01-ilom.sci.example.com datapp1node01-ilom 173.16.6.137 datapp1node02-adm.sci.example.com datapp1node02-adm 173.16.6.143 datapp1node02-ilom.sci.example.com datapp1node02-ilom 173.16.6.138 datapp1node03-adm.sci.example.com datapp1node03-adm 173.16.6.144 datapp1node03-ilom.sci.example.com datapp1node03-ilom 173.16.6.139 datapp1node04-adm.sci.example.com datapp1node04-adm 173.16.6.145 datapp1node04-ilom.sci.example.com datapp1node04-ilom 173.16.6.140 datapp1node05-adm.sci.example.com datapp1node05-adm 173.16.6.146 datapp1node05-ilom.sci.example.com datapp1node05-ilom 173.16.6.141 datapp1node06-adm.sci.example.com datapp1node06-adm 173.16.6.147 datapp1node06-ilom.sci.example.com datapp1node06-ilom ... ... ### DataApp Public Hostnames 192.168.10.152 datapp1node01.sci.example.com datapp1node01-priv.sci.example.com datapp1node01-master.sci.example.com datapp1node01 datapp1node01-priv datapp1node01-master 192.168.10.153 datapp1node02.sci.example.com datapp1node02-priv.sci.example.com datapp1node02 datapp1node01-priv 192.168.10.154 datapp1node03.sci.example.com datapp1node03-priv.sci.example.com datapp1node03 datapp1node03-priv 192.168.10.155 datapp1node04.sci.example.com datapp1node04-priv.sci.example.com datapp1node04 datapp1node04-priv 192.168.10.156 datapp1node05.sci.example.com datapp1node05-priv.sci.example.com datapp1node05 datapp1node05-priv 192.168.10.157 datapp1node06.sci.example.com datapp1node06-priv.sci.example.com datapp1node06 datapp1node06-priv ...   We see a lot of network interfaces because DA is using Network bonding (combination of network interfaces on one host for redundancy and/or increased throughput), but from the outputs and DA documentation we can determine that we have: A 1GbE administration network (eth0). A 10GbE network for external access and communication (bondeth0). A superfast Infiniband backplane network attached to the DA nodes for internal communications (bondib0). As a reminder, 1 and 10 Gigabit Ethernet technologies can, in theory provide respectively a speed of 1 and 10 Gigabit per second whereas a 4 lanes QDR Infiniband standard could provide up to 40 Gb/s throughput. Each network has different IPs and hostnames. Looking at the previous outputs and running commands like hostname -I and hostname -A will help you to determine which IP/hostname relates to our network interfaces.     Here is what we have in our DA environment : datapp1node01.sci.example.com> hostname -I 173.16.6.136 192.168.10.152 192.168.10.100 12.37.250.251 datapp1node01.sci.example.com> hostname -A datapp1node01-adm.sci.example.com datapp1node01-adm datapp1node01.sci.example.com datapp1node01.sci.example.com datapp1node01.sci.example.com>   We notice that the hostname -A command returns only two different hostnames when we have three different kind or IP addresses corresponding to different network interfaces and subnets (173.16.6.136/16, 192.168.10.152/22, 12.35.150.151/16).   This is a very important factor!. It is because the same datapp1node01.sci.example.com sci.example.com hostname is resolved differently if we are inside or outside of the DA collection. On each of the DA nodes the /etc/hosts file is used to resolve the hostname to the private IP address. But if we are outside of the DA machines, we rely on the network DNS.   Microsoft Windows (Version 10.0.16299.1565] (c) 2017 Microsoft Corporation. All rights reserved. C:\Users\RaphP\ping datapp1node02.sci.example.com Pinging datapp1node02.sci.example.com [12.35.150.152] with 32 bytes of data: Reply from 12.35.150.152: bytes=32 time=118ms TTL=61 Reply from 12.35.150.152: bytes=32 time=118ms TTL=61 Reply from 12.35.150.152: bytes=32 time=118ms TTL=61 Ping statistics for 12.35.150.152 Packets : Sent = 3, Received = 3, Lost = 0 (0% loss), Approximate round trip times in mill-seconds: Minimum = 117ms, Maximum = 118ms, Average = 117ms             With the help of the ping command we have confirmed the hostname/IP address associations. We can now draw a small table to describe our multi-NIC configuration for the 5 DA machines.   Network Interface Type Associated IP Address Associated hostnames eth0 1 GbE Admin access 173.16.6.136 173.16.6.137 173.16.6.138 173.16.6.139 173.16.6.140 datapp1node01-adm.sci.example.com   datapp1node02-adm.sci.example.com  datapp1node03-adm.sci.example.com  datapp1node04-adm.sci.example.com  datapp1node05-adm.sci.example.com   bondeth0 10 GbE Client access 12.35.150.151 12.35.150.152 12.35.150.153 12.35.150.154 12.35.150.155 From the outside: datapp1node01.sci.example.com datapp1node02.sci.example.com datapp1node03.sci.example.com datapp1node04.sci.example.com datapp1node05.sci.example.com bondib0 Superfast InfiniBand network for internal communications 192.168.10.152 192.168.10.153 192.168.10.154 192.168.10.155 192.168.10.156 From the DA nodes: datapp1node01.sci.example.com datapp1node02.sci.example.com datapp1node03.sci.example.com datapp1node04.sci.example.com datapp1node05.sci.example.com   It is recommended to draw a similar table when you are facing such a multi-NIC situation at the your Viya site. This single reference point can help prevent confusion. It will be helpful in the Viya pre-configuration phase.   A New Network Configuration Paradigm in Viya   Here is a summary slide for the networking options introduced with the release of Viya 3.5.   The details of each possible parameter and way to implement them is available in the official Deployment Guide.   Also one very cool thing is that : rather than using hard coded IP addresses (SAS_BIND_ADDR or SAS_EXTERNAL_BIND_ADDR) to define the internal and external services binding, you can use “…_IF” variables (such as SAS_BIND_ADDR_IF or SAS_EXTERNAL_BIND_ADDR_IF) set to a network interface name (like “eth0” or “eth1”).   Using the “…_IF” variables can in some (but not necessarily all) use cases, at host machine boot time, see the services pick up the network address of the host corresponding to a specific network interface. This may prove beneficial if dynamic IP addressing is present.   If you are preparing for a Viya configuration, I strongly encourage you to read the official documentation and a related SAS Communities article by my colleague @RobCollum    The First Case Study   With the DA case study, which has three network interfaces, still fresh in our mind, let’s see how we would prepare the deployment of SAS Viya 3.5 in such environment. FYI, The DA infrastructure discussed here is used by SAS teams for internal technical exploration.   Based on the network interfaces table above, here is the first configuration we originally used for the first DA node (datapp1node01.yml): --- network_conf:   SAS_HOSTNAME: datapp1node01.sci.example.com #[internal hostname]   SAS_BIND_ADDR: 192.168.10.152 #[internal network interface Infiniband]   SAS_SAN_DNS: " datapp1node01.sci.example.com localhost" #[certificate DNS field]   SAS_SAN_IP: "127.0.0.1 192.168.10.152 192.168.10.100 12.35.150.151" #[certificate IP field]   SAS_EXTERNAL_HOSTNAME: datapp1node01.sci.example.com #[external  hostname]   SAS_EXTERNAL_BIND_ADDR_IF: "bondeth0" #[external network interface]   We can see that we want to use the private InfiniBand network IP address for the Viya internal communications (192.168.10.152), while the client network (bondeth0) should be used for the external access.   We prepared a similar content (adjusting IP address and hostname) for all the nodes in the datapp1node0<n>.yml files. The deployment of Viya 3.5 using these settings was successful!   Everything was working well… except that CAS was not running…which is pretty critical! 😊 Checking the CAS controller log (caslaunch_default_controller_daemon.log) we noticed this message : ERROR: The TCP/IP tcpSockConnect support routine failed with error 111 (The connection was refused.). ERROR: Failed to connect to host datapp1node01.sci.example.com ', port 5570. 2019-12-17T19:34:47 datapp1node01.sci.example.com:0: ...sas/viya/home/SASFoundation/misc/casluaclnt/lua/swat.lua:193: Could not connect to '..xfhqAXbXPJwTtIKs.' on port 5570.   After a CAS restart the same error message appeared again…   Can you see what was the issue was?   Well…when we think about it, the error message makes perfect sense.   As we can see below, while the CAS nodes try to access the CAS service using hostnames (defined in /opt/sas/viya/config/etc/cas/default/cas.hosts) that resolve to a 192.168.x.x IP address, the CAS service (as instructed via the  SAS_EXTERNAL_BIND_ADDR_IF parameter) is binding port 5570 on the external network  bondeth0 (IP Address: 10.37.150.151).   [root@ datapp1node01 default]# ping datapp1node01.sci.example.com PING datapp1node01.sci.example.com (192.168.10.152) 56(84) bytes of data. 64 bytes from datapp1node01.sci.example.com (192.168.10.152): icmp_seq=1 ttl=64 time=0.024 ms 64 bytes from datapp1node01.sci.example.com (192.168.10.152): icmp_seq=2 ttl=64 time=0.025 ms ^C — datapp1node01.sci.example.com ping statistics — 4 packets transmitted, 4 received, 0% packet loss, time 3311ms rtt min/avg/max/mdev = 0.022/0.023/0.025/0.003 ms [root@ datapp1node01 default]# netstat -anp | grep 5570 tcp 0 0 10.37.150.151:5570 0.0.0.0:* LISTEN 12918/cas   Remember that in this specific configuration, the same hostname: datapp1node01.sci.example.com  is used for the internal hostname (resolved locally through /etc/hosts) and the external hostname (resolved by the DNS) – this is the reason why our initial configuration was not working !   There are three approaches outlined below, which could be considered to useful in this situation. However the first approach Is generally preferred/recommended in many situations.   In case we’d like to have the CAS front door service listening on all network interfaces, the recommended/default option  would be to set the SAS_EXTERNAL_BIND_ADDR variable to 0.0.0.0 . That way the CASL port (5570) would be reachable whatever network interface (admin, ethernet, or Infiniband) is used : [cloud-user@intcas01 ~]$ sudo netstat -anp | grep cas tcp        0      0 192.168.2.1:43806       0.0….0.0:*               LISTEN      22893/cas tcp        0      0 0.0.0.0:5570            0.0.0.0:*               LISTEN      22893/cas tcp        0      0 127.0.0.1:41222         0.0.0.0:*               LISTEN      22893/cas tcp        0      0 192.168.2.1:43806       192.168.2.2:35838       ESTABLISHED 22893/cas tcp        0      0 192.168.2.1:43806       192.168.2.2:35840       ESTABLISHED 22893/cas   Removing all together the SAS_EXTERNAL_ parameters in the network.conf files and redeploy Viya. In such case, CAS can start but the CAS service binds port 5570 to the IP address corresponding to the SAS_BIND_ADDR value (and therefore can only accept connections from the inside – like from another DA node). This may not be appropriate when we have client applications on a different subnet trying to connect to CAS e.g. 3 rd  party programming interfaces. Another and probably better option could be to use one of the internal hostname aliases defined in the /etc/hosts files to identify uniquely the internal hostname (without having the name conflict with the external hostname). In our use case with the DA there was a uniquely defined internal name i.e.  datapp1node01-priv.sci.example.com example for the first machine (datapp1) :   --- network_conf:   SAS_HOSTNAME: datapp1node01-priv.sci.example.com #[internal hostname]   SAS_BIND_ADDR: 192.168.10.152 #[internal network interface Infiniband]   SAS_SAN_DNS: " datapp1node01.sci.example.com localhost datapp1node01-priv.sci.example.com " #[certificate DNS field]   SAS_SAN_IP: "127.0.0.1 192.168.10.152 192.168.10.100 12.35.150.151" #[certificate IP field]   SAS_EXTERNAL_HOSTNAME: datapp1node01.sci.example.com #[external hostname]   SAS_EXTERNAL_BIND_ADDR_IF: "bondeth0 #[external network interface]   That way we could keep CAS listening on the external network interface and accept connections from CAS clients such as 3 rd  party programming interfaces.   The Second Case Study   In this second case study, the topology is pretty simple: 1 host machine with CAS and 1 host machine with the Viya services layer (microservices and Infrastructure servers) and SPRE.   In the lead up to the installation it was noted that each of the customer’s host machines had 2 NICs (internal and external).   The diagram below shows what the customer wanted to achieve:     Here is the table corresponding to this specific customer situation, showing the network interfaces, IP addresses and hostnames corresponding to the 2 Viya servers.   Network Interface Type Associated IP Address Associated hostnames eth1 10 GbE Internal network 192.168.254.100 192.168.254.101 int2005.gs.example.com int2008.gs.example.com eth0 External Client access 123.123.1.190 123.123.1.191 ext71.gs.example.com ext76.gs.example.com   It’s recommended to document as much detail regarding the networking configuration as possible to avoid any misunderstanding. From this information we prepared the network.conf files as below (first the first machine): --- network_conf:   SAS_HOSTNAME: int2005.gs.example.com #[internal hostname]   SAS_BIND_ADDR_IF: eth1 #[internal network interface 10GB]   SAS_SAN_DNS: " int2005.gs.example.com ext71.gs.example.com localhost" #[internal hostname, external hostname, localhost]   SAS_SAN_IP: " 192.168.254.100 123.123.1.190 127.0.0.1" #[internal IP, external IP, 127.0.0.1]   SAS_EXTERNAL_HOSTNAME: ext71.gs.example.com #[external hostname]   SAS_EXTERNAL_BIND_ADDR_IF: eth0 #[external network interface 1GB]   Once again, the deployment was successful but there was an issue: the SASLogon microservice was failing showing errors message related to the TLS certificates :   Certificate for <...> doesn't match any of the subject alternative names [...]   After a bit of digging, and running some openssl commands, we discovered that on the machine where the Viya services and infrastructure servers (int2005) had been deployed, the Apache httpd had been installed with the certificates already configured.   But unfortunately, the certificates SAN (Subject Alternative Name) did NOT include the internal name (see below an extract).   Hostname :CN= ext71.gs.example.com SAN:     X509v3 Subject Alternative Name:          DNS: ext71.gs.example.com, DNS: www.ext71.gs.example.com   There was no reference to the internal int2005.gs.example.com name in the certificate.   Now the SASLogon error message was making sense…the microservices were not able to contact the apache HTTPD server using the internal hostname because the HTTPD TLS certificates was only valid for the external hostname.   So, at this point, there was two possible solutions: Either have the admin add the internal hostnames in the certificate (because internal services connect to the same HTTP server to communicate altogether) Use a front-end proxy where the custom cert can be used and leave the default certs in our own Viya Httpd server (see  this post from @EdoardoRiva for instructions). Finally, after some discussions, the customer’s system administrator was able to add the internal hostname in the TLS certificate SAN.   After running the deployment, a second time with the updated TLS certificates in place, everything was working as expected in the Viya environment    Limits and Potential   While the new network configuration options provide a lot of flexibility, it’s important to note the caveats and provisos: Currently there no automated checks for validating the network.conf files If you want to configure your multi-NIC deployment for 10 machines, then you must prepare 10 different YAML files and place them in the host_vars sub-folder of the playbook directory, adjusting the values for each of them. If you are not careful it is very easy to make a mistake. In our case study, during one of the attempts there was a typo: SAS_EXTERNAL_BIND_ADD_IF, instead of SAS_EXTERNAL_BIND_ADDR_IF (!) The installation ran fine, no errors, no warnings, nothing. But the Viya environment was not functioning as expected. Configuration values not kept across multiple attempts You might have noticed that when you run multiple deployments from the same playbook folder, the log file (deployment.log) and common configuration files (like inventory.ini and vars.yml) are kept in the snapshot folder of the playbook directory. However, the network configuration files used for a specific deployment are not kept. So, it is important to keep track of the values you use for a specific deployment in case of multiple attempts. SAS_EXTERNAL_* variables only used for binding services SAS_EXTERNAL_* variables allow us to configure the public Viay access points to bind on the specified network interfaces, but they don’t replace the CAS and HTTP services configuration variables used to deal with an external proxy or load-balancer. Since an additional 3 rd party reverse proxy that’s been placed in front of Viya’s Apache HTTP proxy service is not managed by these Viya network.conf variables, you still need to use configuration parameters such as SERVICE_BASE_URL, CAS_VIRTUAL_HOST or sas.httpproxy.external.hostname/port (as explained this post by @EdoardoRiva. Refer to the official documentation.   Even with these caveats, this entirely new paradigm for network configuration with the support of network interfaces and Classless Inter-Domain Routing (CIDR) range addresses, is likely to help in some scenarios deal with planned or unplanned changes in the IP addresses. At this time (March 2020) it is important to note that the Viya deployment guides will still refer to the requirement for static IP addresses.   Conclusion As we’ve seen, many factors (multiple IP addresses and hostname resolution, HTTPD TLS certificates, etc…) might impact the way a multi-NIC Viya deployment should be prepared. Make sure you have a good understanding of them.   Hopefully the 2 case studies provided, give an idea of the challenges you may face and how the new networking configuration options can be used to navigate them successfully.   A “Big Thanks” to my colleagues at SAS who helped in their expertise and interactions to provide the content of this post.       ... View more

As you probably know, a new version of SAS Viya was shipped in late November of 2019.   As a reminder, SAS Viya is on its fifth major release. Version 3.1 Fall 2016 Version 3.2 April 2017 (announcement at SGF) Version 3.3 December 2017 Version 3.4 July 2018 (18w30), 2019 May Version (19w21), 2019 August Version (19w34) Version 3.5 November 2019 (19w47) It is worth noting that the successive SAS Viya 3.4 releases (May and August) also brought many changes. So users who have only been using the initial release of SAS Viya 3.4, will only discover them when starting to use SAS Viya 3.5.   19w47 is a major ship event as it brings new versions of many Viya products and solutions whether for the Data management (SAS Data Preparation 2.5), in the Models/Decisions Management area (SAS Model Manager 15.3, SAS Intelligent Decisioning 5.4) or in the analytic space (SAS Visual Analytics 8.5, SAS Visual Statistics 8.5, SAS Visual Data Mining and Machine Learning 8.5, SAS Visual Text Analytics 8.5, SAS Visual Forecasting 8.5, SAS Optimization 8.5, SAS Econometrics 8.5 and SAS IML 8.5). Each of the new versions or products brings its own whole set of exciting new features.   But this article will focus on Architecture and Deployment topics and has two purposes: Firstly, to list and highlight some of the key changes from an Architecture perspective in this new release and provide some pointers to additional resources for further information. Secondly to zoom in on some specific technical details for Viya 3.5 which will likely impact the work of the SAS Administrator/Installation Engineer. However, it is far from being an exhaustive list of all the changes coming with Viya 3.5. If you want to have more information on Viya 3.5 new features, a lot of resources have been made available for you: See the General What's new page, and the products specific what's new information. Support of Power-9 Architecture     SAS Viya 3.5 brings IBM POWER9 processor chip architecture support running Linux. This often referred to as the PowerLinux platform (PLX). SAS now supports Viya on Linux for Intel x64, AMD and Power 9 architectures.   SAS Viya can now take advantage of the new highly performant IBM POWER9 servers (including GPU utilization for Deep learning algorithms).   However, there are a some caveats at this point in time: Customers wishing to license Viya 3.5 for this platform will be required to discuss with the SAS sales team their specific needs to ensure they can be met, as not all Viya based offerings are currently available on the IBM POWER 9 platform as of January 20th, 2020 The current supported offerings as of January 20th, 2020 are: VA / VS / VDMML / Model Studio / VTA / Data Prep / Data Agent / Econometrics / VF / Optimization / Intelligent Decisioning / Model Manager / CONNECT / SPRE - QC / SPRE - IML / Risk Modeling Nodes for Visual Data Mining / Access Support: PC Files, ODBC and Postgres Accessing certain data structures like Parquet are currently not supported SUSE is not supported for use with Viya 3.5 on Power 9 servers In terms of deployment, the PLX Platform needs to be chosen for the order and the orchestration CLI must use the new flag "–architecture" to specify the Power Linux architecture. Finally, a specific Red Hat Linux operating system, taking advantage of the Power Linux processors, must be installed. SAS Job Flow scheduler   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SAS Viya 3.5 introduces the Job Flow Scheduler that you can use to create a chain of actions composed of: code (SAS or other languages), scripts (bash, shell, etc.), user interface actions (import files and tables with SAS Data Explorer), run data plans (SAS Data Studio). You can add logic gates (OR, AND), create time-based triggers or schedule a flow. See Bogdan Teleuca’s recent article for a detailed explanations about this new product. SAS Job Flow Scheduler is a licensed offering like other offerings from SAS.   Until Viya 3.5, it was only possible to create flows through the command line, but SAS Job Flow Scheduler at Viya 3.5, a graphical user interface makes this feature much easier to use.   Support for additional network configurations   In previous releases of Viya, there have been some deployment challenges with Viya in scenarios where there are more complex or specific network configurations (hosts with multiple IP address/hostnames, IP address in Viya services auto-generated certificates, need to override default hostname, etc...).   A scenario where this was observed was when customer environments had host machines with 2 or multiple network interface cards (NICs). With Viya 3.5, SAS provides: A better way to handle more sophisticated network configurations like multiple IP Addresses, multiple Hostnames For example, when there is one network for internal communications and another for external access, we can force the Viya internal services to bind only on the internal network). A consistent way across all the Viya services to use a set of defined values for the IP addresses and hostnames in the internal, external communications and generated TLS certs. Instead of having the Viya Services relying on potentially inconsistent results from hostname –f or hostname –A ad-hoc commands (which may or may not report the wanted value), network configuration settings can be explicitly set for each machine of the deployment. There are 3 categories of settings: IP addresses used with SAS Viya TLS Certificate Parameters (to affect TLS certificates “Subject Alternative Name” value) Binding variables for external ports (variables for Public facing services such as HTTPD, CASL, etc…) Before the deployment you must define your network settings in one file per host machine in the deployment playbook “host_var” folder (refer to the documentation for detailed instructions).   This is very flexible, but as with all environments where there are multiple configurable components, the choice of the proper configuration option will require careful planning. We also recommend customers test their environments using their own use cases to ensure their configured Viya 3.5 environment behaves in the required manner. Read carefully the official documentation to fully understand what you are doing. Reduced Memory footprint Java Spring Microservices are used in Viya 3.5. In the development of Viya 3.5, consolidation of several services were completed to improve efficiency.This has helped reduce the memory footprint and for some services improved execution times.   Postgres 11 and HA Pgpool The Viya 3.4 version of the SAS Infrastructure Data Server is based on PostgreSQL version 9.4 that is reaching End-of-Life on February 13, 2020. By itself it could already be a good reason to consider an upgrade to Viya 3.5 that is shipping with Postgres 11.   A new Viya 3.5 deployment will install Postgres 11. A Postgres upgrade utility (from Postgres 9.4 to 11) for those customers upgrading from Viya 3.4 to Viya 3.5 is provided.   The utility called sas-pgupgrade-cli automatically scans a SAS Viya deployment for all running PostgreSQL databases and migrates them to the latest version of PostgreSQL that was installed by SAS on the physical machines.   Also, with the official support of Pgpool in HA mode, the availability of the Viya 3.5 platform is increased compared to previous versions of Viya.   As a reminder Pgpool is the load-balancer in front of the PostgreSQL cluster. You can expect to see additional articles from my colleagues talking about Pgpool in the context of (higher) availability during 2020. Using SAS Studio? Then read on.     With Viya 3.5 there is only one version of SAS Studio (for Viya): 5.2. However it is important to note there are 2 ways which SAS Studio can leveraged.   The first is the "Basic Edition" approach (which relies on the object spawner and workspace server technology). The second is the "Enterprise Edition" (which works with the Viya Microservices and the Compute and Launcher servers).   Programming Only deployments come with the "Basic Edition" while a full-deployment of Viya gets the both versions (Basic and Enterprise).   It is recommended that in a full deployment of Viya 3.5, the end-users should only use the "Enterprise Edition". A good practice is to disable SAS Studio "Basic Edition" in a full Viya deployment.   This new SAS Studio 5.2 release brings many cool improvements such as Job definition authoring with prompting, local file system navigation and access, Query builder, Background submit, Scheduling, DATA Step Debugger, Git integration with UI. For more details, see this recent article by Edoardo Riva. Hadoop 3 support Current Hadoop distributions (such as Cloudera 6.x, Hortonworks HDP 3.x, or MapR 6.x) are now using the Apache Hadoop 3 base code and as the end-of-support (EOL) date draws nearer, customers running Hadoop 2 based environments, might start to look at a migration project to version 3.     Hortonworks Support policy (REVISED October 15, 2019)   Hadoop 3 brought many changes and started to be supported with the May release (19w21) of Viya 3.4.   There is, now a Hadoop section in the "Support for database" documentation and it’s great to see that both CAS Data Connectors (for serial and multi-node loading) and CAS Data Connect Accelerators (for parallel loading) as well as SAS Plug-ins for Hadoop and SAS Scoring Accelerator for Hadoop are now officially supported with Hadoop 3 based distribution.   However, there are 2 remaining caveats that I’ll remind here: The Embedded process have always used the HCatalog component to manage the hadoop structures. As this component was dropped by Hortonworks in HDP 3.x, working with the SAS Embedded Process and Hive Managed tables in HDP 3.x requires LLAP (Hive Live Long and Process) to be turned “on” in the Hadoop cluster. It is critical that SAS administrators are aware of this when migrating to Hadoop 3. The SAS plug-ins for Hadoop incudes a security fix using HDFS block tokens. It worked well with Hadoop 2.x distributions but with Hadoop 3 there was a breaking change in the HDFS block tokens API. While it is in the process of being fixed, a deployment of the SAS Plug-ins in a secured "Hadoop 3 based" distribution (such as Cloudera 6.3.2 or Apache Hadoop 3.2.1) is likely to require HDFS block token to be disabled for CAS access. CASLibs Data source enhancements There are several great enhancements in terms of data integration for the CAS Server. New ADLS CASLib With SAS Viya 3.5, Azure Data Lake Storage is a new data source: CAS supports adding a caslib to Azure Data Lake Storage Gen2 for data access.   The ADLS caslib type supports loading and saving delimited files (CSV files) and Optimized Row Columnar (ORC) files. Multiple HDFS cluster support CAS can now assign CASLibs to multiple remote HDFS clusters on a "caslib-by-caslib" basis (see the documentation).   It requires SAS plug-ins for Hadoop to be deployed on each remote HDFS cluster but it means that the same CAS instance can use HDFS caslibs with both co-located and remote HDFS cluster. New file types support As a nice picture is usually a better way to explain than a long text, I'll reproduce here a table created by my colleague Nicolas Robert that gives a good summary of the changes coming with Viya 3.5.     For more details see Nicolas Robert's article.   The orange triangle signals what's new with SAS Viya 3.5. Security The main changes in Viya 3.5 regarding the authentication are: Support of Kerberos constrained delegation (allows to support environments with Windows Defender Credential Guard enabled) Support of service accounts with the Compute Server (equivalent of the default sassrv account in SAS 9); full support of Kerberos in Multi-tenant environments. Zoom in on the deployment experience Aside from the major change which is the network settings, there were a few changes that I found in my experience, to be worthy of a mention in this article  Let’s share! A new Viya ARK version with new behaviors SAS Viya ARK provides automated tools and utilities to help SAS customers prepare for a SAS Viya deployment. But the most useful that every SAS Viya Installation engineer/administrator should know are the Pre-installation of SAS Viya System Requirements and the SAS Viya Multi-Machine Services Utilities – aka "MMSU" (recommended way to stop and start Viya services in Multi-Machines deployments).   While it is not supported directly through the SAS Technical Support, SAS Viya ARK is a GitHub project and the GitHub model can be used for tracking bugs and feature requests through GitHub issue or pull request for support.   There is a new Viya ARK release for Viya 3.5 with several changes.   Several issues have been fixed but there are also changes in the pre-installation playbooks behavior and consultants well accustomed to the 3.4 version could be taken off guards. SELinux Configuration With Viya 3.4-ARK SELinux was automatically set to permissive to allow the deployment   With Viya 3.5-ARK, if the mode is set to "enforcing", the pre-install playbook will fail and a message directing the user to take the necessary actions will be given.   The rational behind this change is that the requirement to disable SELinux was "relaxed" (since the Viya 3.4 August 2019 version). There are ways to cope with an environment which cannot disable SELinux or even turn it off during the installation, and they are documented in the deployment guide. So if Viya ARK detect enforced SELinux, then instead of disabling it, it will stop and print a message for the user to look at the documentation on how to configure SELinux to allow the deployment. Supported Java Runtime Environment (JRE) Versions A minimum version of Java is now required (1.8.0_121 at the time of the writing) .   If there is no Java installed, Viya ARK will install it.   BUT if Java is already installed in previous version, the check mode fails but  the normal mode does not update the Java package (a debug message will show the version found to be installed).   So, if Java is already installed in the target environment check mode is highly recommended. Don't rush the validation ! With Viya 3.4, generally as soon as the deployment was completed, it was possible to logon and start the validation process.   However, in some of our Viya 3.5 deployment tests we noticed that the environment was not always immediately ready just after a successful deployment (failure=0 reported by each host of the deployment in the site.yml playbook run).   When connecting to the Visual interfaces too quickly after the deployment, a user could see missing GUI elements or dialog like:   In such case, just be patient and give it another try a few minutes later.  What happens is that the jobs that populate Environment Manager view behind the scenes, run on a schedule, and those jobs must complete before a user could see anything. Of course this behavior was observed in our own virtualized environment and might not occur with a more performant infrastructure (in terms of CPU and RAM). Merged services While merging microservices had a positive impact in reducing the overall memory footprint needed for a Viya deployment, the change has broad impact for upgrades and for administrators who are familiar with the existing services.   As explained in the usage note in the Administration guide, some services have been merged into logical groups, in Environment manager, there is now a new concept of “Host service”. If a service has been merged with other services, there is a way to see it in Environment Manager (follow the official documentation instructions).   But when starting or stopping a “merged” service, you need to use the “Host service” name.   For example, the CAS Proxy service and the CAS Management service have been combined into a CAS Administration service. If the CAS Proxy service is down, then the CAS Administration service must be restarted.     Another side effect is for upgrade situations, because the log files of the old services are not cleaned up (discussions are underway to determine the possibility of developing a playbook in Viya ARK to do this post-upgrade clean up). Conclusion While SAS Viya 3.5 has just been released with a lot of improvements and new features, both functional and from a technical architecture and deployment perspective.   Thanks for reading! ... View more

Since early October a new SAS Viya QuickStart has been made available for Google Cloud Platform. TL;DR This tool is a great fit if you want to quickly deploy a "ready to use" SAS Viya 3.4 environment in the Google Cloud Platform without having to go through the process of Cloud provisioning (choosing machines, storage, load-balancers, configuring networks, firewall rules, etc...) and Viya pre-requisite and deployment.   Read carefully the  README file provided in the GitHub project and after some configuration, you can provision and deploy a full Viya environment in GCP by running a single command in the Google SDK command-line.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   If you want to better understand how the QuickStart works and learn from real hands-on experience, keep on reading...:)   As it turned out to be a longer article than expected, here is a table of content below that you can use to jump directly to the topic of your interest: Google Deployment Manager The Architecture The Prerequisites Some tips to increase your chances of success Important things to note Conclusion Google Deployment Manager There are many ways to automate the provisioning and deployment of an application or a Software in the Google Cloud Platform (aka GCP).   One can use automation tools which have connectors to Cloud provider (such as ansible) or specialized IaaC (Infrastructure as a code) tools such as Terraform.   But what we provide in the QuickStart is a "Google Deployment Template" for the Google Deployment Manager which is the official way to go and is documented by Google.   This GitHub project is the main source of instructions and documentation for the Viya Quick Start: https://github.com/sassoftware/quickstart-sas-viya-gcp   Let's have a look at the architecture, pre-requisites  and see what it takes to use the QuickStart to deploy and run Viya in the Google Cloud Platform. The architecture The diagram below comes from the GitHub project README file.     The QuickStart creates the GCP infrastructure for you. The QuickStart focuses on a subset of the Viya offerings and available topologies available to customers who have licensed SAS: A single CAS SMP Server (in the diagram it's termed "CAS Controller", but to be clear it performs analytical processing) Single Viya services server Only a small set of supported products (VA/VS/VDMML 8.3 and higher) The SAS Data agent is currently not supported in this QuickStart. The deployment also includes a local openLDAP server (Instructions are provided to create new users). Post-deployment instructions are provided to: replace Self-Signed TLS Certificate with custom certificate enable access to existing Data Sources (for SAS/ACCESS engines) It is also important to note that currently, there is no single point and click license agreement with Google & SAS, for running SAS Viya in GCP. Therefore, there is no contractual support from Google. And whoever uses the QuickStart is responsible for the costs of the GCP resources that are consumed by its deployment. The Prerequisites A GCP account with access to a GCP project A SAS software order Email The SAS Deployment Data zip file needs to be placed in a GCS bucket (if you have no idea of what a GCS bucket means, no worries. There is an explanation further down in the blog). During our tests, we discovered that there was an additional level of requirement relating to the GCP account. This relates to the IAM role.   The service account XXXXXXXXXXXX@cloudservices.gserviceaccount.com, associated to the GCP project, needs to have the "Project IAM Admin" role.   This role is required to allow the necessary privileges to execute the "setIamPolicy" command that is used by the QuickStart template.   The user account you are using to do that (like john.smith@mycompany.com) needs to have the "Owner role" on the Google Cloud project. So, for a customer it is someone who likely plays the role of the administrator of the project. How easy is the Deployment process? The deployment process, itself is straight forward:   1) Grab a "jumphost" machine from where you can install git then the GCP SDK (gcloud commands) and configure it to work with a Google Cloud Project (the "jumphost" can be your own laptop, a VM or a container running on your laptop or a machine in Openstack)   2) Clone the "Viya for GCP QuickStart" project from GitHub   git clone https://github.com/sassoftware/quickstart-sas-viya-gc   3) Edit a few settings in a configuration file (instance size, GCP Zone ,etc...)   4) Run a single command to deploy the template.   Then wait 1 to 2 hours and BADABOUM! You have a brand new Viya deployment up and running in the Google Cloud!   Looks nice and easy, right?   But...as with other deployments, the need to pay attention to the detail is critical. Why? Well...behind these 4 simple steps, there are some instructions that might not appear so clear if you are new to the Google Cloud Platform. For example, you might have missed or misunderstood one of the values required in the configuration or maybe you did not know how to prepare the deployment data file in the GCS bucket, or your deployment was successful but you are not able to open the "SAS Logon" page, because you made a mistake in the Ingress rules definitions...hopefully the next section will be helpful.   Some tips to increase your chances of success How to upload the order deployment data in the GCS bucket To run a Viya platform in GCP, you need to make a license file available for it, right?   For that you must make the license available in the Google Storage services, called "Google Cloud Storage" (equivalent of S3 in AWS).   In the GCP console, find the Storage service and click on "Browser".     Just click on "Create Bucket", give it a name.   Then, in this new bucket, you can upload the "SAS_Data_Deployment.zip" file that is attached to your Software Order Email confirmation (which contains the license itself).     Finally, if you click on the uploaded file, you can see the URI.     Of course, you could automate all these steps by simply running a few Google SDK commands (like gsutil cp ).   Copy the URI value. This is what you will need in the configuration file (for the DeploymentDataLocation variable). Example of configuration settings This is the most important step for the GCP quick start. If it is correctly set, then there should not be any issue with your deployment. Let's have a look as this file.     The README file has a nice table which provides some help to understand what we are supposed to provide for the configuration variables.     However, some parameters involve a little more work than others. GCP zone.   Although it does not appear in the above Configuration table, you also need to provide the GCP deployment zone.   For example:   Zone: us-east1-b   If you leave the default value (GCP_ZONE), the deployment will fail.   SSH Public key.     The QuickStart creates a Bastion machine in GCP from which you can orchestrate the other machines.   If you don't have one already, create a key-pair with a command like this:   ssh-keygen -q -t rsa -N '' -f ~/.ssh/frapovqs   The command creates 2 files in your ".ssh" folder: frarpovqs and frarpovqs.pub which respectively contain the private and the public key.   The public key needs to be configured in the GCP VM (and the QuickStart does it for you) so you can connect with SSH using the private key as your credential.   We need to copy the content of the .pub file and set it as the value of the SSHPublicKey varaiable in the configuration file, it should look like in the example below:   SSHPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDY9wbxpRk19YwPtjYJOyyRlrUwDR9WiVLRD/obbY887JpPd5DDSsgdy4/t+KVUu8x8Ma8XiXG7aeP6iAr+kXmnag7/Gn1cFd4AbjUmjaoWNZIg3tsCWiWt2xo14nSzu980Py3VvXX8/mu681bLQfA7MdjlroAMDI4y2xWneldJhPELe02HrQHNtT/5feWiPDfmgdD+0AdjfyF/eFDna3TFXjkp5tVRrSnLf46IIzHwntTLawBiG/DSBs45G7r31AbEqT7Yaiz1pn48CbD5qzddNhQXFlmTvUnAJgzMJpvZuIXaRc6WDm45wPMSGphR8zpddPgrG8eMpQKOHxsO/7xH centos@frarpo-jumphost.dept-gbc.central4-r1.infraserv1.sas.com   Make sure the value is on one single line.   Deployment Data Location   It corresponds to the "SAS_Data_Deployment.zip" file that we placed. We need to provide the URI of the file in GCS.   For example:   DeploymentDataLocation: gs://frarpo-viyaquickstart/SAS_Viya_deployment_data_9C7SRH.zip   DeploymentDataLocation: gs://frarpo-viyaquickstart/SAS_Viya_deployment_data_9C7SRH.zip   AdminIngressLocation and WebIngressLocation   These parameters allow the opening or restriction of network access from the outside to the environment that gets deployed in GCP.   As an example, here are the values I used:   AdminIngressLocation: 149.173.8.44/32 # Allow inbound HTTP(S) traffic to the Web Server from this CIDR block (IP address range). Must be a valid IP CIDR range of the form x.x.x.x/x. WebIngressLocation: 149.173.8.65/32   As I used the range "32", it means that only the specified IP address can can connect to the Bastion host in GCP.   For AdminIngressLocation it corresponds to the external IP of the jumphost machine in Openstack that I'm using to drive the QuickStart deployment. Be careful. The IP that is reported locally by the jump host does not necessarily correspond to the value that is required here. Remember that our deployment is in the Public Cloud on Internet. So we need to use the IP of the jump host as known from the Internet.   For example:   [centos@frarpo-jumphost ~]$ curl https://www.whatismyip.org/ | grep "Your IP" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<h3 class="text-center global-ip"><strong>Your IP:</strong> <a href="/my-ip-address">149.173.8.44</a></h3> 100 36377 0 36377 0 0 74097 0 --:--:-- --:--:-- --:--:-- 74087   For WebIngressLocation it corresponds to the IP of my own workstation (also returned by a web site such as whatismyip.org).But a different CIDR range can also be used if one wants to open to multiple IP addresses.   If you are not so comfortable with the CIDR IP range notation (with the "/") you can use this handy calculator: https://mxtoolbox.com/subnetcalculator.aspx# to understand how many IP addresses correspond to a specific CIDR IP range. One ring command to rule them all... Once everything is ready in the configuration file, just run the magic command:   gcloud deployment-manager deployments create frarpo-deployment --config ~/gcpquickstart/quickstart-sas-viya-gcp/templates/sas-viya-config.yaml --async   When you run this command, after 2 seconds you see:   The fingerprint of the deployment is iayJnj_Es9_eA-hP0D9raQ== NAME TYPE STATUS TARGET ERRORS WARNINGS operation-1572527013070-596347a03863e-7c1fc568-999c9dff insert RUNNING frarpo-deployment [] [] [centos@frarpo-jumphost ~]$   If it is the first time you do it, it's worth monitoring the deployment...because it may fail (see below for my own troubleshooting experience). If you are confident and did it before with success, you can go for a walk and come back 1 or 2 hours later. Monitor the deployment progress You can monitor what's going on with the GCP web console:     You can click on it to have more details on the deployment progress from the Google cloud perspective.   Now if you want to see deployment logs from a Viya perspective, follow the instructions in the README file to select the ansible controller VM and see the  "Serial port 1" log:   What if it does not work? The README file provides some Troubleshooting guidelines.   Here is a little feedback from my own troubleshooting experience...As many people have said "learning from failures is the best way to learn"     Click on the link provided in the error message:     And enable it.   Then retry.   Another issue that I faced was related to missing privileges for the service account.     After adding the right role or privilege to the GCP project service account (" Project IAM Admin" role " as explained in the pre-requisite section), the issue was resolved.   A successful deployment will look like this:     Note: The warnings come from the fact that the OS image used by the QuickStart has been fixed to a specific version to avoid a known issue (http://support.sas.com/kb/64/703.html) of Viya 3.4 and specific versions of systemd in Linux. How to access the Viya environment? As explained in the GitHub README you can find the IP address to access your newly deployed environment by browsing in the GCP console (Network Services then Load-balancing, advanced view)     Assuming you have the correct information, you should be able to login with the sasadmin account and see something like:     Otherwise you will see something like:     This error likely means that the network security configuration (created by the Viya QuickStart deployment) does not allow you to access the load balancer.   In such case, go in the "Network Security", then "Cloud Armor" and check if the access rule match with your local IP address, like below:     Check out https://www.whatismyip.com/ if you are not sure what your IP address is).   Finally, you can also access though ssh, from your jumphost machine with the sasinstall account to the ansible controller machine, like:   ssh -i ~/.ssh/frapovqs sasinstall@35.196.188.241 Important things to note You can use a mirror   There is a parameter for it in the configuration file (otherwise the ansible controller will instruct GCP VMs to download the SAS packages from the SAS Hosted repository machines on Internet)   Where are my users?   An openLDAP server is installed as part of the QuickStart deployment, the Viya service GCP VM will get the OpenLDAP playbook (thanks to Erwan Granger) from https://github.com/sassoftware/quickstart-sas-viya-common and run it to deploy and configure openLDAP and also SSSD.   However, the environment could then be configured to point to a real LDAP server (such as the customer corporate Active Directory server) as long it is available from the GCP VPC network.   The README file provides instructions on how to add new users.   Use the right GCP project   When you run the single command to perform the deployment in GCP, you don't need to specify explicitly state your Google account and Google project, because it is already configured in the Google SDK configuration.   But if you are working with multiple Google projects, you have to be careful because you need to point to the project where you plan to deploy Viya.   For example:   Show current project:   gcloud config get-value core/project sas-gelsandbox   Switch project:   gcloud config set project sas-gelsandbox Updated property [core/project].   List VM instances in the current project:   gcloud compute instances list   Don't forget to remove/clean-up the environment when you don't need the it anymore   The great thing with the Google Deployment Manager is that all the resources created by a deployment are considered as a single entity, the "deployment" object. So if we want to get rid of the deployed environment and all associated resources (network, load-balancer, accounts, etc...) it is very easy, you just need to run a command like:   gcloud deployment-manager deployments delete frarpo-deployment Conclusion The QuickStart is great to quickly stand-up a pre-defined Viya environment in GCP with the proper cloud provisioning (service accounts, machines, networks, load-balancer).   Once you have used it, you can save some "cloud money" by removing it (and all associated resources) as easily as you created it and then whenever you need it again, recreate it with the same single command.   As it is provided as free source code in GitHub, the QuickStart can also be a great baseline for anyone who would like to extend it for more complex deployments or to build its own Viya IaaC framework.   Many thanks to Dante Bergamasco for his help and support in the write up of this article.   Thanks for reading! ... View more

You might have missed it; but, if your SAS user groups within your organization have begun using the Google Cloud, there was an important new offering released in 19w34 which may help them. This new offering is SAS/ACCESS engine for Google BigQuery.   In this article, I'll cover some key points, briefly introduce Google BigQuery, show how to implement the connection from SAS and CAS to BigQuery, and try to answer some of the typical questions a technical architect/integration specialist would ask. Key points SAS/ACCESS interface for Google BigQuery is available for SAS 9 (9.4M6 or later) and Viya (although a 64bit linux is the required OS for both) and provides SQL Pass-Through Facility and Bulk-Load Support features The required Google BigQuery client library is included with SAS/ACCESS Interface to Google BigQuery. The SAS/ACCESS Interface to Google BigQuery includes SAS Data Connector to Google BigQuery which means that, in a Viya deployment, CAS can directly access to the BigQuery tables (and also that in a CAS MPP deployment multi-node data loading is available). Here is the official SAS documentation.   What is Google BigQuery ?  BigQuery is an enterprise data warehouse provided as a Google Cloud Platform service. According to Google, BigQuery can process billions of rows in seconds. It is scalable and has an in-memory engine which provides various analytical capabilities. Once the data has been moved into BigQuery, you can run queries using the good old SQL language leveraging the processing and architecture features of Google's infrastructure.   It is very easy to load and query Big Query either using the SQL UI (available in the GCP console) or using the command line (if the Google Cloud SDK has been installed).   The BigQuery documentation provides quick-start for both method: using the Web UI or the command-line tools.   You first need to define a "Dataset", in which you create your tables from scratch with some SQL code or import the data by uploading them from your PC or from the server where the "bq" command line is installed. You can also import from another GCP services like Google Cloud Storage (AWS S3 equivalent), Google Drive or BigTable.   BigQuery also has a "Data Transfer Service" with various connectors and agents that can be used to migrate data from other SaaS providers, Cloud storage solutions as AWS S3, or Cloud or on-Prem Data warehouses (such as Amazon RedShift or Teradata) into BigQuery.   For the purpose of this article, we loaded the names.csv file using the "upload" method pointing directly on a web resource from this datasets catalog and called our BQ table: "babynames".   Once you have loaded your data, you can query them either from the BigQuery Web UI Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   or with the "bq" command line:   [centos@frarpo-jumphost ~]$ bq show sas-gel:sampleds.babynames Table sas-gel:sampleds.babynames Last modified Schema Total Rows Total Bytes Expiration Time Partitioning Labels ----------------- --------------------- ------------ ------------- ------------ ------------------- -------- 27 Aug 12:11:42 |- BRTH_YR: integer 13962 789099 |- GNDR: string |- ETHCTY: string |- NM: string |- CNT: integer |- RNK: integer   The BigQuery engine exposes a REST API for easy programmatic access and application integration. Client libraries are available in Java, Python, Node.js, C#, Go, Ruby, and PHP. And finally, there are a variety of third-party tools that you can use to interact with BigQuery, such as visualizing the data or loading the data.   Before trying out the example below, it's worth taking a moment to discuss the costs if using Big Query.    As of October 7th, 2019 BigQuery charges for data storage, streaming inserts, and for querying data. Either with the on-demand option: based on the amount of data processed by each query you run Or with a predefined capacity and a flat-rate But loading and exporting data are free of charge. Also note that the service is free for up to 1 TB of queried data per month and 10 GB of stored data. Finally, Google provides a convenient cost calculator where you can estimate the costs specifically for BigQuery.   Hopefully – you now have a rough idea of what BigQuery is, but for more details about the BigQuery, the best information source would be the official Google Cloud page.   Of course, Google is not the first one to offer this type of service in the Cloud sometimes called "DBaaS" for "Database As A Service" (you might have heard about Amazon Redshift or Azure SQL Datawarehouse?)   Now let's see how SAS can work with it! Now how do I access the data from SAS? BigQuery is a "serverless" Cloud Datawarehouse. It means that when you want to connect to it, you don’t have to provide a server name and port, or even a user ID.   You are simply contacting a specific database service that is made available for your Cloud tenant (which is called a “project” in GCP).   Now the question is: how do we authenticate?   While several authentication methods exist in the Google Cloud Platform to call the GCP APIs , Google usually recommends to use a service account that is associated with the GCP project (as opposed to a specific user).   In addition, our current version of SAS/ACCESS interface to Google BigQuery only supports authentication with a service account (oAuth authentication is in the roadmap for future releases of SAS Viya).   So, let's see how to do this authentication with the service account.   If you have the appropriate privileges, you can use the GCP console to create a private key for the service account attached to your Google project and download the related credentials as a JSON file.       A private key will be stored inside the generated JSON file along with the service account details (email, id, project id, etc…). Several JSON files (each containing a different private key) can be generated for several or the same service account and used to gain access to the APIs.   As explained in the SAS documentation, what we need to provide to run a successful SAS LIBNAME statement for the Google BigQuery Engine is: a path to the file containing the service account credentials (.json format) the Google project ID. Assuming we have made the .JSON file (containing our service account credential) available on the machine hosting the Compute Server, we should be able to run something like:   LIBNAME BQ bigquery CRED_PATH='/home/viyademo01/bq/sas-gel-ae7285ebb812.json' PROJECT='sas-gel' SCHEMA='sampleds';     It is not working...   But it is because, for the first time we access, we need to enable the API.   So, if we open the provided URL ( and make sure we are signed in with the google account associated to your project) we see something like:     If we click on the "Enable" button and try again to run our SAS Libname statement:     Hurrah! It is working now !   Now, we can see the table's columns and open it, just like with any other SAS/ACCESS interface.   Testing the Data Connector To allow CAS to access directly to the BigQuery data, we need to create a "bigquery" type caslib.   Here is the code to create a Google BigQuery caslib:   cas mysession;caslib gbq desc='Google BigQuery Caslib' dataSource=(srctype='bigquery' credfile='/opt/sas/viya/bqdc/sas-gel-ae7285ebb812.json' project='sas-gel' schema='sampleds');   The key difference is that, this time, the credential file must be available on the CAS Controller as that's the place where access to BigQuery is done.   Also, if you are using the Viya visual interfaces or SASStudioV, remember that (by default) the CAS actions run under the "cas" account identity. So, it is required that the credential file ("sas-gel-ae7285ebb812.json" in this example) is located in a folder where the "cas" account can read it.   Otherwise (for example, if you place it a user personal folder) you will get a message like:   ERROR: The connection to the data source driver failed. ERROR: Error: stat /home/viyademo01/bqdc/sas-gel-ae7285ebb812.json: permission denied ERROR: Function failed.   List and Load data from Google BigQuery table.   proc casutil; list files incaslib="gbq"; load casdata="babynames" incaslib="gbq" outcaslib="casuser" casout="names_from_gbqc"; list files incaslib="casuser"; contents casdata="%upcase(names_from_gbqc)" incaslib="casuser"; quit; Can I load the data in parallel from BigQuery into CAS? So strictly speaking: No.   Because what is generally called "parallel loading" refers to the Embedded process load mechanism where all the distributed database "nodes" send or get data in parallel across multiple CAS workers.   Currently (October 2019), it is only possible to do that with Teradata, Hadoop or Spark as they are the only data system for which we offer the SAS In-Database Technology product on SAS Viya.   However, in a CAS MPP deployment, what you can do with the SAS/ACCESS engine for Viya and the Data Connector is to perform “multi-node loading.” It means that each CAS worker will access to a slice of the data. Although this technique does not guarantee parallel transfer from either side, it can be a more flexible solution than serial loading. Workload communications will be spread across all the CAS workers (instead of simply using the CAS controller as a single point of interaction).   For information about the loading options available to users of SAS Viya, please take a look at the excellent 2019 SGF paper written by Rob Collum of SAS.   So, let's see what how to make it happen.   Here is the code:   cas mysession; caslib gbqmn desc='Google BigQuery Multi-Node Caslib' dataSource=(srctype='bigquery' credfile='/opt/sas/viya/bqdc/sas-gel-ae7285ebb812.json' project='sas-gel' schema='sampleds' NUMREADNODES=10 NUMWRITENODES=10); proc casutil; list files incaslib="gbqmn"; load casdata="babynames" incaslib="gbqmn" outcaslib="casuser" casout="names_from_gbqmn"; list files incaslib="casuser"; contents casdata="%upcase(names_from_gbqmn)" incaslib="casuser"; quit;   The only change is the addition of NUMREADNODES and NUMWRITENODES (please refer to Uttam Kumar's article if you want more details on CAS Multi-Node data loading). However, if you only do that: no matter how many CAS workers you have, you will see a message saying that only one worker was used (Actually it is the CAS controller doing the standard serial load):   WARNING: The value of numReadNodes(5) exceeds the number of available worker nodes(1). The load will proceed with numReadNodes=1.   The trick here is that : we need to make the credential file available and readable by the cas account in the same path (/opt/sas/viya/bqdc/sas-gel-ae7285ebb812.json) on all the CAS workers. So, you can either make the credential file available through a shared file system location or copy the credential file in the exact same location on all CAS worker hosts and make sure the CAS account can read it.   When you do that, the log shows a message confirming the multi-node loading:   87    proc casutil; NOTE: The UUID '92291f20-72e5-4344-8032-e0606a0d2191' is connected using session MYSESSION. 88       /*list files incaslib="gbqmn";*/ 89       load casdata="babynames" incaslib="gbqmn" outcaslib="casuser" 90            casout="names_from_gbqmn"; WARNING: The value of numReadNodes(10) exceeds the number of available worker nodes(2). The load will proceed with numReadNodes=2. NOTE: Cloud Analytic Services made the external data from babynames available as table NAMES_FROM_GBQMN in caslib CASUSER(viyademo01). NOTE: The Cloud Analytic Services server processed the request in 32.891526 seconds. 91       /*list files incaslib="casuser"; 92       contents casdata="%upcase(names_from_gbqmn)" incaslib="casuser";*/ 93    quit; NOTE: PROCEDURE CASUTIL used (Total process time): real time           32.90 seconds cpu time            1.02 seconds   Now we are sure that the data loading is done in multi-node mode. Are my data encrypted? Well... it is not really a question for us (SAS Consultants), but more for Google as the encryption has to happen between the BigQuery client and the BigQuery engine (there is no option to control that from the SAS/ACCESS interface perspective).   But with a bit of research, here is what we can learn from the official Google documentation: By default, BigQuery encrypts customer content stored at rest using Google managed keys. However, you can decide to use customer managed encryption keys.   Regarding encryption in transit, although there is no detailed documentation on how it is specifically implemented with BigQuery, the official documentation provides this information regarding “data in transit”:   "To protect your data as it travels over the Internet during read and write operations, GCP uses Transport Layer Security (HTTPS). For more information, see Encryption in transit in GCP.   Within Google data centers, your data is encrypted when it is transferred between machines." See this whitepaper for more details. What if I want user-based access control? While it is possible, in BigQuery to set permissions and predefined or custom Cloud IAM roles, the current version of the SAS/ACCESS engine only supports service account authentication.   Therefore, there is no way to implement permissions at the end-user level when accessing with the SAS/ACCESS engine.   However, it is possible to define multiple service accounts within the same google project with different predefined or even custom roles (associated to permissions such as dataset or tables creation, deletion, read/write privileges).   It is also interesting to note, that, to date, with Google BigQuery, you cannot grant permissions on specific tables, views, columns, or rows. A dataset is the lowest level resource that supports access controls in BigQuery. Conclusion As more companies and organizations look to reduce the total cost of ownership (TCO) for their existing data warehouse implementations, the public Cloud based database offerings (Amazon Redshift, Azure SQL Datawarehouse, BigQuery), Cloud object storage (AWS S3, Amazon Data lake Storage, Google Cloud storage) or SaaS (Snowflakes, etc...) are becoming an attractive and viable option(s).   It is good to see SAS developing new ACCESS engines data connectors and CAS native access (caslibs) to ensure that our customers can continue to leverage the power of SAS analytics when they decide to move their data in the Cloud.   Thanks for reading and good luck with using SAS/ACCESS to BigQuery. ... View more

As explained by Mark Thomas in his comprehensive article, SAS Viya 3.3 introduced improved integration with SAS Event Stream Processing 5.1 components and products.   In a nutshell, SAS Event Stream Processing (ESP) is the SAS solution to run event stream processing models in real time, a very high-throughput and very low latency engine with streaming analytics capabilities.   The objective here is to revisit Mark’s article with the new SAS ESP 5.2 version and SAS Viya 3.4 integration and also to touch on some important changes brought by this latest ESP version from an Architecture and Deployment standpoint: docker deployment for ESP on the Edge, security changes, etc.   ESP products If you are new to ESP, you might be a bit lost at first and confused by the various ESP products and components.     There are only a few changes compared to Mark’s article for SAS ESP 5.1. We still have 4 ESP products, but they have different licensing rules and deployment modes and platforms.   Let’s recap!!!     Hopefully, this table will help to quickly identify the key differences between the products in terms of licensing, deployment mode, and platform support. However don't take it for granted for always. The licensing rules are not static and may continue to evolve.   Deployment options Until SAS ESP 5.1, there was a difference in a “stand alone” deployment of ESP compared to what you would do when ESP was a part of a larger SAS Viya order.   With SAS ESP 5.2, no matter if ESP comes alone or bundled with another SAS Viya solution (VA/VS/VDMML, etc…), the deployment will be done with an ansible playbook and the ESP components will be integrated with several SAS Viya Services.   Add-on products such as SAS Event Stream Processing for CAS, SAS Event Stream Manager (ESPCLUSTR), or SAS Event Stream Manager (ESPMGR) are also deployed via the ansible playbook.   If you are familiar with SAS Viya deployment with ansible, the table below can be useful to understand the SAS Viya host groups and services that come with the ESP products.     So even the stand-alone version of ESP comes with all the SAS Viya services required to the ESP-Viya integration (only the items in bold correspond to the ESP core capabilities).   Therefore: ESP Studio and ESM will now always use the SAS Infrastructure Data Server (PostgreSQL) by default to store their content. (Note that Streamviewer continues to use the small footprint H2 database in SAS ESP 5.2.) Authentication in ESP Studio and ESM, like any other SAS Viya web application, is now done through the SAS Logon Manager. ESP Studio and ESM end-user’s identities are now managed through the SAS Viya Identities microservice. The SAS message broker allows other applications to communicate with ESP application (notifications when a model is selected as a champion in Model manager). The "Host Group placement" and "HA" considerations explained in Mark Thomas’s article remain valid and are now true for any type of ESP Server installation (either stand-alone or integrated in SAS Viya). I encourage you to read them if you are working on the architecture design of your ESP solution. The only case where the deployment method is different is the ESP for the Edge computing.   ESP for Edge computing deployment SAS Event Stream Processing for Edge Computing requires a separate order and is a smaller footprint version of SAS ESP to allow it to run on edge devices such a sensor, routers, etc.   The table below shows the differences in terms of components included in ESP Edge versus the full ESP. Also, the SAS Viya services and infrastructure servers are not part of the ESP on the Edge deployment (which significantly reduces the footprint).   ESP for Edge computing is also quite unique for a SAS product. In the current licensing policy SAS® Event Stream Processing for Edge Computing is licensed under a perpetual license structure. The customer pays a one-time fee to purchase a given quantity of the software and owns it forever. However, annual support and maintenance is a separate charge and has an annual fee (But again the licensing rules are subject to chaneg in the time).   With the 5.2 release, the deployment of SAS Event Stream Processing for Edge Computing can be done in 2 ways: standard Linux deployment or Docker.   To ease the roll-out of ESP on edge devices, it is convenient to have a "central software repository" with the ESP packages or image, so the remote devices can simply pull, deploy, and run ESP in an automated way.   This central repository will be either a package mirror or a docker registry with the ESP docker images.   Standard Linux deployment on the Edge The first thing to notice is that you have more options in terms of Linux flavor. Because this product’s aim is to install and run ESP on edge devices: sensors, routers, etc, more options are provided to support a wider range of edge devices.   ESP deployment on devices with ARM processors (the most widely used architecture in mobile devices) is also supported but only with Debian Linux or Ubuntu Operating Systems.     A mirror repository is required for SAS Event Stream Processing for Edge Computing and will play the role of the "central software repository."   The "edge_mirror.sh" script provided as the "Mirror Extension for SAS Event Stream Processing for Edge Computing for SAS Viya 3.4" (see screenshot below) will orchestrate the mirror manager to create a specific Edge repository and place packages in separate directories (basic, analytics, astore, gpu, etc).     Finally, the customer can selectively deploy packages to the edge devices using specific edge package managers (rpm, deb). The linux packages can then be installed (respecting the package dependencies) using the customer preferred way (rpm, apt-get). Note that it is up to the customer to manage a potential automation of the pull and install procedure on the local edge device.   Docker deployment on the Edge If the customer is running Docker on the edge devices, it can greatly simplify the deployment and its automation. Nevertheless, keep in mind that the docker version comes with some restrictions compared to the Linux version (no GPU support, no ARM support).   Instructions are provided in the "ESP for Edge Computing" deployment guide to create a local docker registry that can be used as a centralized repository to deploy ESP on edge devices. Once it is in place, all you must do to start ESP on the Edge is to run a single command to "pull" and start the docker image. Note that two flavors of the docker image are available, "esp-edge-base" or "esp-edge-analytics" (includes SAS ESP Analytics and ASTORE support). The "base" image size is 750 MB, whereas the size of the "with analytics" one is around 1.5 GB.   Security   A key change for the security topic in SAS ESP 5.2 is the way to configure encryption and authentication for ESP servers and clients with the new Security Configuration file. This file is read by the ESP server at startup time to determine if the ESP is secured (TLS encryption for communication from ESP clients) and, if ESP Server authentication is in place, which authentication method is used.   Here is an extract of this configuration file:   Select image to see a larger version.   Thanks to the improved ESP and SAS Viya integration, "SASLogon" will almost always be an option for ESP Server authentication (even the stand-alone installation comes with SAS Logon manager).   Finally, another new feature is the support of an additional authentication method: OAuth 2.0 Cloud Authentication and Authorization, which can be very useful for integration in PaaS Cloud platform such as Cloud Foundry.   Note that this authentication method (unlike the existing OAuth method) requires a “live” UUA server to validate end-user credentials in real time.   Conclusion This article is far from being exhaustive on architecture and deployment changes in SAS ESP 5.2. It is simply an overview and provides considerations for some of them. However, if you are looking for more details about the architecture, deployment, and security with SAS ESP 5.2 then stay tuned. A new SAS ESP VLE with deployment hands-on will soon be available.   Thanks for reading! ... View more

With SAS Viya 3.4 comes "Mirror Manager" which significantly eases the build and use of local packages repositories, aka "Mirrors." In this article, I will try to convince you to always plan, build and use a Mirror as part of your SAS Viya installation and also give you some cool tips with Mirror Manager.   As a reminder, during the deployment of SAS Viya, the ansible playbook instructs the machines, that are part of the deployment, to download the SAS packages (RPMs or MSI) from the "internet facing" SAS Hosted repository servers.   This "mirror" term refers to the fact that we are building a copy of the original SAS Packages Repository. Then the deployment can download the SAS packages from this copied repository instead of using the SAS Hosted one.   I remember the time when I was downloading the SAS 9 Depot on my USB Hard Drive, before visiting a customer for a deployment. You know...just in case he had forgotten to download it in preparation of our meeting... As the guy in charge of the installation I liked to have this security.   Do you miss that? Don’t worry ... you can still do it with Mirror Manager.   ALWAYS Build a mirror - why? Here are some of reasons why you should build and use a mirror: The usage of a mirror is required, anyway in several cases. when there is no internet access on the SAS Viya deployment machines for SAS Viya deployment in SUSE Linux environments for Multi-Tenancy or multiple CAS Servers Using a mirror allows you to reinstall the exactly same version of the packages. Your mirror is static, whereas the SAS Hosted repository is dynamic and is always be updated with the latest versions of the packages. So, if you want multiple environments with the exact same version of the software it is the way to go. Using a local repository reduces the security risk exposure of Internet access. Once your local mirror is in place and available in the customer internal network, it allows to install on servers that has no access to Internet. You can build it anywhere! One of the target machine of the deployment A shared File or Web Server at customer site, Your own laptop, etc... And on any kind of operating system: Windows, Linux and even Mac.   ALWAYS Build a mirror - how? With SAS Viya 3.4 and Mirror Manager it is now, very simple to build and use a mirror.   Step 1 - Build it All the things you need to build your mirror are: The deployment data zip file from the Software Order email (typically: SAS_Viya_deployment_data.zip) Mirror Manager utility that you can freely download. A machine (can be your laptop or one of the servers you plan to install) with enough disk space. See documentation   Once you have downloaded and unzipped Mirror Manager on your local machine or server, open a shell (DOS or Linux bash depending on your host operating system) and run Mirror Manager utility to ensure it is working, for example:   C:\MirroMgr>mirrormgr help   Mirror manager is a command line utility and can do many things, through 4 types of commands:   Click any image to see a larger version.   The most interesting command for us, though is "mirror". This command is the one we use to build our own mirror of the SAS Packages repository.   The "--help" option of the command gives us more details on how to use it :     So, for example if we want to build a mirror for our SAS Viya order, all we need is to specify the deployment data zip file (SAS_Deployment_data.zip) that was attached the Software Order email:   C:\MirrorMgr>mirrormgr mirror --deployment-data 3.4se18w30\SAS_Viya_deployment_data_18w30.zip   If we do that Mirror Manager starts to create the local repository in the user's home directory (C:\users\frarpo in my case). However, we might not have enough space there to store a repository. Depending on the type of order, the size of the repository (containing all the packages for your ordered product) can vary from 13 to 50 or more GBs.   So, it is much better to specify an alternate location where you've made sure that you had enough space (your USB Hard drive for example).   C:\MirrorMgr>mirrormgr mirror --deployment-data 3.4se18w30\SAS_Viya_deployment_data_18w30.zip --path "e:\Repo34se18w30\"   But hold on ... what will be exactly downloaded if I do that?   Well, mirrormgr will download all the packages corresponding to your order but for all types of available platform – it means: RedHat, SUSE and potentially others in the future.   So, we likely don’t need to have the SAS packages for any type of platform, right?   Fortunately, we have the "list remote" command that can help here, it lists the available options:   C:\MirrorMgrTests>mirrormgr.exe list remote platforms --deployment-data 3.4se18w47/SAS_Viya_deployment_data_18w47.zip x64-redhat-linux-6 x64-redhat-linux-7 x64-suse-linux-12   However, don’t get confused by the number at the end, it is not the Linux version. Use x64-redhat-linux-6 for SAS Viya deployment on all supported versions of Red Hat Enterprise Linux and its equivalent such as Oracle Linux. Use x64-suse-linux-12 for SAS Viya deployment on all supported versions of SUSE Linux. Don't use x64-redhat-linux-7 as it refers to something different (docker repository).   So finally, the command you need to run is:   C:\MirrorMgr>mirrormgr mirror --deployment-data 3.4se18w30\SAS_Viya_deployment_data_18w30.zip --path "e:\Repo34se18w30\" --platform x64-redhat-linux-6 --latest   Note: we add the "--latest" option to ensure that Mirror Manager download the latest verions of the SAS packages (including the Hotfixes)   Now, we can see that Mirror Manager is starting to download the packages from the SAS Hosted repository:   Mirroring 1/2 repos/shipped/va/104/va-104-x64_redhat_linux_6-yum 118.44 MiB / 13.53 GiB [>-------------------------------------------------------------------] 0.91% 1.09 MiB/s 3h27m5s   So what do we get at the end? Well, we get an organized structure that contains the SAS software packages themselves (RPMs or MSI or other depending on your products) and some associated metadata files.   See below an example of the mirror repository content and structure:     Step 2 - Serve it Once the download is complete, you need to make your mirror (or rather your local package repository) available for the machines where you plan to deploy SAS Viya.   The cool thing is that you can archive and move the content of your newly created "repo" directory wherever you want.   Then there are multiple methods to make it available for the deployment: Copy it everywhere (on all SAS Viya target machines) NFS-Mount it everywhere (on all SAS Viya target machines) Serve it over http or https The principle of the local repository is to have a single location for the packages, that all the SAS Viya target server can access to download the packages that they need. So usually we want to make this location available through an HTTP server or through the file system (A shared file system is used when we have multiple machines in the deployment).   Step 3 - Use it The last thing to do is to ensure that the deployment playbook will use our brand new local mirror as its package repository.   All the SAS Viya target hosts need to be able to access the mirror location, but thanks to Ansible, we only have one file to edit to set the location: vars.yml.   So before running the ansible deployment playbook, check the value of REPOSITORY_WAREHOUSE in your vars.yml, and replace the default value with the location of your local mirror.   REPOSITORY_WAREHOUSE: "https://ses.sas.download/ses/"   We can see that by default the access to the repository is done through https (because the communication with our SAS Hosted repository server must be secured) but it can also be done through standard HTTP or even File system access like:   REPOSITORY_WAREHOUSE: "http://sasviya01.race.sas.com:8123/"   REPOSITORY_WAREHOUSE: "file:/nas/dept/mis/viya/sas_repos/test/"   REPOSITORY_WAREHOUSE: "file:///c:/sasinstall/mirrorViyaWindows18w47" (for windows deployment)   Or   REPOSITORY_WAREHOUSE: "file:/sasinstall/mirror18w47"   See RFC for examples of file URI.   But remember that the repository warehouse URL must be available to all hosts that participate in the deployment since the hosts are going to use that address to retrieve packages from the repositories. For example, if the repository warehouse is file-based, then that location should be shared across hosts and should be shared at the same path on each of those hosts.   You can stop your reading here unless you’d like to know more and learn some tips.   There are many useful things that you can do with the mirror manager. Let's review them quickly.   Tip 1 - Build a mirror from another mirror By default, the Mirror Manager tries to download the packages from the SAS Hosted repository (typically "http://ses.sas.download"). However, using the "url" option, you can direct the Mirror manager to build the local mirror from another mirror.   It can be very useful for lockdown customer environments where a "jumphost" or "bastion" is used to get the packages, but cannot be directly accessed by the machines participating in the SAS Viya deployment.   The diagram below illustrates this use case:     Example:   c:\viya34tmp>mirrormgr mirror --deployment-data C:\viya34tmp\SAS_Viya_deployment_data.zip --path C:\viya34tmp\MyRepo --platform x64-redhat-linux-6 --latest --insecure --url "https://gelweb.race.sas.com/mirror/09MYXC/"   Note: if your serverA, in the scenario above, only has access to Internet through an HTTP or HTTPS proxy, mirror manager can deal with it. See the official documentation.   Tip 2 - Check the mirror integrity Mirror Manager has a "verify" command that you can use to check your mirror repository content. "Verify" checks the package hash in the repository metadata against the file on disk, verifying that the packages on disk have not be tampered with and are all present. Note that you need to ensure to use the same latest and platform flags when running verify against the mirror as when it was created.   Example:   c:\MirrorMgrTests>mirrormgr.exe verify --path "3.4se18w30/myRepo34se18w30" Verifying 1/2 repos\shipped\va\104\va-104-x64_redhat_linux_6-yum 13.53 GiB / 13.53 GiB [=====================================================================] 100.00% 441.28 MiB/s 31s Verifying 2/2 sasmd\shipped\va\104\va-104-x64_redhat_linux_6-yumsasmd 2.28 MiB / 2.28 MiB [=========================================================================] 100.00% 44.92 MiB/s 0s Repository repos\shipped\va\104\va-104-x64_redhat_linux_6-yum: verified Repository sasmd\shipped\va\104\va-104-x64_redhat_linux_6-yumsasmd: verified   Tip 3 - Use Diff to see the new packages Now, you are interested by a potential software update and you’d like to see if your mirror could be refreshed to benefit from the latest packages. Mirror Manager can help you!   You can use the "mirror diff" command:   mirrormgr mirror diff --deployment-data path-to-deployment-zip-file-from-SOE --path path-to-mirror-destination --latest   Let's see an example:   C:\MirrorMgrTests>mirrormgr mirror diff --deployment-data 3.4se18w30\SAS_Viya_deployment_data_18w30.zip --path 3.4se18w30\myRepo34se18w30\ --latest --platform x64-redhat-linux-6 Starting diff process Diffing 1/4 repos/shipped/va/100/va-100-x64_suse_linux_12-yum Diffing 2/4 repos/shipped/va/104/va-104-x64_redhat_linux_6-yum Diffing 3/4 sasmd/shipped/va/100/va-100-x64_suse_linux_12-yumsasmd Diffing 4/4 sasmd/shipped/va/104/va-104-x64_redhat_linux_6-yumsasmd Done Repository sasmd/shipped/va/104/va-104-x64_redhat_linux_6-yumsasmd: no new packages found Repository repos/shipped/va/100/va-100-x64_suse_linux_12-yum: new packages found New package: sas-tkmtrb1-01.12.00-20180711.230148391635.suse.x86_64.rpm New package: sas-baseui1-03.13.00-20180711.211149548837.suse.x86_64.rpm New package: sas-tkspdsdrv1-03.13.00-20180711.231639760650.suse.x86_64.rpm New package: sas-tkair-03.13.00-20180711.233006399589.suse.x86_64.rpm New package: sas-relationships-3.6.16-20180613.1528913768264.suse.x86_64.rpm New package: sas-tkgraph2-03.13.00-20180711.224145354350.suse.x86_64.rpm ... <more and more lines> ... New package: sas-cpp-birdprint6-6.1.105646-20180905.1536177913.sad New package: sas-certframe-3.1.46-20180625.1529949596423.sad New package: sas-licenses-cli-1.4.7-20180603.1528064123282.sad New package: sas-tkfnc1-03.13.00-20180711.223804780708.sad New package: sas-compsrvcfg-03.11.00-20180711.212231034110.sad New package: sas-crsmtpsrvc-01.13.00-20180711.220101186130.sad New package: sas-tklauncher1-01.13.00-20180711.225810925646.sad New package: sas-rabbitmq-server-3.7.3-20180515.1526408034975.sad New package: sas-tsmodel1-03.13.00-20180711.232208269444.sad New package: sas-tkqc1-03.13.00-20180711.230820989006.sad New package: sas-backup-agent-2.1.29-20180612.1528836411326.sad New package: sas-grodscrendr1-3.9.0-20180711.232222463982.sad New package: sas-themecontent-1.1.62-20180711.1531289140877.sad New package: sas-tkaacas-03.13.00-20180711.232854757131.sad New package: sas-audit-cli-1.5.0-20180601.1527863137426.sad New package: sas-crsstat-01.13.00-20180711.223739333395.sad New package: sas-casluaclnt-01.13.00-20180711.210139577646.sad New package: sas-crsfcmpact-01.13.00-20180711.214312253840.sad New package: sas-tkaccess1-03.13.00-20180711.222916806727.sad New package: sas-tkstat1-03.13.00-20180711.231715741134.sad New package: sas-ygforcast-1.13.0-20180711.232941482699.sad New package: sas-cpp-commons6-6.1.105646-20180905.1536177913.sad New package: sas-transformations-2.4.53-20180615.1529111481300.sad New package: sas-crossdomainproxy-1.3.2-20180619.1529398991726.sad New package: sas-orchestration-cli-1.1.42-20180816.1534448442316.sad New package: sas-jobexecapp-2.1.20-20180612.1528837342703.sad New package: sas-search-2.1.37-20180613.1528914066291.sad New package: sas-tkdmine1-03.13.00-20180711.223458429223.sad New package: sas-jobexecution-2.4.22-20180615.1529092031764.sad New package: sas-tkhps1-03.13.00-20180711.224743011776.sad New package: sas-docconvjars-1.0.12-20180525.1527279525342.sad New package: sas-securedom-03.13.00-20180711.231147153078.sad New package: sas-tkspde1-03.13.00-20180711.231448585680.sad New package: sas-qkbs-cli-1.3.3-20180603.1528065858373.sad   That's a very long list! And I'm surprised because I'm using an order that I created yesterday, so I should not have many new packages, right? (It might change in the future but AFAIK SAS RnD hasn’t yet fully implemented the "continuous delivery" approach with several updates within a day.)   Okay, so if we have a better look we now see that there is some extra information that we don't really care about: we see the "suse" packages (even if we filtered with the platform type) and also, we see the "sad" files which are not very relevant as we are only interested in the new packages.   This issue might have been fixed in the more recent version of mirror manager. Otherwhise what we can do is "kind of" filtering the output using "grep -v" or "findstr /v" to exclude lines that are matching "sad" or "suse" pattern - as in the example below.   C:\MirrorMgrTests>mirrormgr mirror diff --deployment-data 3.4se18w30\SAS_Viya_deployment_data_18w30.zip --path 3.4se18w30\myRepo34se18w30\ --latest --platform x64-redhat-linux-6 | findstr /R /v "suse sad" > newpackages.txt   Once you know what the new RPMs are, you can decide (in prevision of a software update of your deployment) to "refresh" your mirror : synchronize it with the SAS repository in preparation of an update of your deployment, with a command like:   C:\MirrorMgrTests>mirrormgr mirror --deployment-data 3.4se18w30\SAS_Viya_deployment_data_18w30.zip --path 3.4se18w30\myRepo34se18w30\ --latest   As explained in the updating your SAS Viya software section of the official deployment guide.   Tip 4 - Check the Mirror Manager logs You have problems using Mirror Manager? But you don’t see any logs? Mirror Manager like all the nice tools logs your actions. By default, the logs are stored in the user application folders. For the linux version of mirror manager it is $HOME/.local/share/mirrormgr For the windows version of mirror manager it is %LOCALAPPDATA%\\mirrormgr   Tip 5 - Use the customer existing mirror You might have noticed this comment in the vars.yml for the repository-warehouse variable:   # # If the SAS packages are already available to the server (e.g. through a RHN satellite channel), # set this variable to "none" to indicate no additional repository is needed for deployment. REPOSITORY_WAREHOUSE: "https://ses.sas.download/ses/"   A very interesting thing with SAS Viya 3.4 is that: if your customer has found a way to make the SAS packages available for the target machines into his existing corporate package repository, you can configure REPOSITORY_WAREHOUSE to make your deployment rely on the customer's repository (instead of using a specific one just for SAS packages).   Tip 6 - Get the deployment playbook from the mirror As you know (if you deployed SAS Viya at least once), one of the very first step of your deployment is to generate the installation playbook corresponding to your software order.   That is the purpose of the SAS orchestration CLI: it uses your Software Deployment data file (SAS_Viya_Deployment_data.zip) as an input to retrieve some deployment data from the SAS Hosted server and build the playbook locally.   But actually, instead of contacting the SAS Hosted server, you now have an option to instruct the orchestration CLI to use your local mirror instead! So now you can do that even if you are not connected to Internet! Here is an example:   .\sas-orchestration build --input c:\sas\install\SAS_Viya_deployment_data.zip --repository-warehouse file:///sas_repos   The cool thing: it also automatically inserts the mirror location value in the vars.yml that is inside your playbook! So you don’t even have to change it – when you run the deployment playbook the target machines directly download the packages from the mirror that you used to build the playbook.   Conclusion Okay, that's it! I hope I convinced you. Now just do it! Plan the mirror in your architecture designs, build it, and use it!   Big thanks to Eliott Peel and Erwan Granger for their contributions to this post.   ... View more

The SAS Viya 3.4 platform was released in the middle of last year (24th of July 2018). Less than 2 month after (18w38), the support of Microsoft Windows for Viya was made available. However many customers still have a Viya 3.3 platform in production or are still assessing the deployment of Viya (either in addition or as a transition target for their existing SAS 9 platform).   So now is a good time to come back on the key new architecture and deployment changes that came with the SAS Viya 3.4 release.   New supported OS (Windows, SUSE)   With SAS Viya 3.4, we introduce the support of two additional Operating System: SUSE Linux and Windows.   However, today the Windows flavor of SAS Viya is only be available for single machine deployments. All the SAS Viya components (Microservices, Web Application, Infrastructure Servers and CAS) need to be deployed on the same host (which obviously implies that only CAS SMP is supported in Windows deployments).   In terms of deployment (binaries installation and configuration), instead of Ansible and YUM packaging, Command/PowerShell scripts and MSI packages is used. A dedicated deployment guide of SAS Viya for Windows is available in the SAS documentation.   Regarding SUSE Linux, we won't have the same "Single Machine" limitation, but the deployment technology is different. We'll continue to use Ansible to orchestrate the deployment tasks, but it will interact with a different package manager called "Zypper" (instead of YUM) in order to lay down the Software binaries on the target machines.   In addition, for SUSE Linux deployments, it is mandatory to create and use a mirror repository for the software packages (instructions on how to use a mirror repository are provided in the official deployment Guide). There are also a few additional system requirements, as the default shell (bash) for the target machine(s) as SUSE and RHEL have sometimes slightly different ways to configure the Linux system or simply use different tools.   Finally, a noticeable change in the deployment process (introduced by the support of the new OS) is that the orchestration CLI is used to generate different "deployment artifact" depending on the target Operating System (ansible playbook or Powershell script).   Mirror Manager   Using a local mirror repository for your SAS Viya deployment is a general best practice for many reasons (remove the internet connection requirement for the target servers, avoid the maximum download trap, keep a static version of the SW packages for consistency across environments or tenants deployments, etc). In addition, in case of a deployment with SUSE Linux, it is mandatory.   However, the process to build and use a mirror for the deployment was not so simple with Viya 3.3. The efforts to improve this process have resulted in a completely new process and a new tool called "SAS Mirror Manager."     It is a "command line" tool that you can run anywhere (on any Linux, or on your local PC with Macintosh or Windows) which allows you to create AND refresh a local mirror repository from the SAS Hosted Repository.   By default, the mirror content will be placed in the "sas_repos" folder of the user's home directory; however, you can point to any directory available from the machine running the tool (such as a web server directory or a shared NFS mount).   Depending on your target Operating System, the tool downloads the appropriate native's packages (RPM files for Linux RedHat and SUSE, MSI files for Windows, APT files for Debian-based Linux, etc).   The key point is to ensure that the default location or the location that you select has enough free space because in the current version of the tool, you will not be able to know in advance how much storage space will be used.   From our experience, around 15 GB would be required for a very simple order (with VA for example), whereas a full stack order repository takes around 60 GB of Disk space.   Once downloaded, it is possible to move the files either in a web server directory (for example: /var/www/html/pulp/) or a shared NFS mount (for example: using secure copy or "rsync" commands).   Once the mirror repository has been made available and can be reached from all your target servers, all you will have to do before running your deployment playbook is to provide the mirror location in the vars.yml file, for example:   REPOSITORY_WAREHOUSE: "http://sascas01.race.sas.com:8123/09XXXX/"   Finally, it is worth noting that with SAS Viya 3.4, the Orchestration CLI can also point to a pre-built mirror repository to generate the playbook from the SAS Deployment Data file (attached in the Software Order Confirmation email).   /opt/sas/viya/home/bin/sas-orchestration build --platform redhat --input ./SAS_Viya_deployment_data.zip --repository-warehouse https://gelweb.race.sas.com/mirror/$ORDERNUM/sas_repos --deployment-type programming --output SAS_Viya_playbook_prog.tgz   New Host Groups, SAS Studio 5.1   As new components have been added with SAS Viya 3.4, new "Host Groups" appear in the deployment playbook inventory.   As an example, for the simplest possible order (in terms of product): Visual Analytics, there are 4 new Host Groups.   The "CognitiveComputingServices" Hostgroup was available with specific products such as Model Manager or VDMML, it is now part of the minimum set of SAS Viya Microservices to perform common text analytics tasks. The "ComputeServer" Host Group is also now systematically provided, not for Visual Analytics (which is exclusively using the CAS engine) but to support the new "SAS Studio Viya" which needs to run SAS Code in SPRE on behalf of the Compute Service. The "GraphBuilderServices" host group contains services that provide tools to create and edit custom graphs. The "StudioViya" Host Group contains the services providing the new SAS Studio 5.1. SAS Studio 5.1 (aka  "SAS Studio Viya") is the new SAS Programming tool generation embracing the SAS Viya architecture paradigm (Microservice based, using REST APIs instead of IOM protocol). It is available by default in every "full" deployment of SAS Viya.   Host groups are very important for the SAS Technical Architect as he needs to decide on which host(s) they will go and whether they are subject to specific placement rules, support multiple instances distributed across machines, fail-over, etc.   Tip: Very useful information is often provided directly in the Host Groups comment of your inventory.ini default file.   Multiple CAS Servers     In SAS Viya 3.3, the only way to have multiple CAS Servers within the same SAS Viya deployment was to have restrictive requirements on the LDAP structure and to deploy the SW in a Multi-tenancy mode.   In the 3.4 release, it becomes possible to have multiple CAS servers, even with a standard single-tenant deployment.   In terms of deployment process, you always start with a single CAS Server (SMP or MPP), "cas-shared-default", but then you can add new empty machines where you can install additional CAS Server (SMP or MPP).   CAS resource management   In the new SAS Viya version, new options are added for a better repartition of the CPU resources used by the CAS processing. In addition, it is now possible with the priority-levels policies to define disk quotas for the loaded tables and allocate a share of the CPU capacity to different groups of users (up to 5 groups).   Those options leverage cgroups which is a Linux Kernel feature, hence they are only available in the Linux version of CAS. As a consequence having the cgroups feature installed on the CAS machines is a new requirement if we want to use those new capabilities.   The resources used by CAS can be managed: Either broadly with cas.MEMORYSIZE (limit the memory usage on the CAS node to a specific amount of RAM) and cas.CPUSHARES (limit the CPU usage on the CAS node to a share of the overall CPU capacity) server configuration options. or more specifically through policies associated to group of users. There are 2 types of policies: Global caslibs: one policy per CAS Server that places a disk cache space quota on global caslibs. Priority-level: one to five policies per CAS Server that place disk cache space quota on personal caslibs and a CPU consumption limit on CAS sessions that can be mapped to group of users. How does it work? At startup time, the CAS server reads the policies defined in Consul and dynamically create the cgroups to limit resources usage (which means that the resource management policies are only working in full deployment of SAS Viya - as the SAS Configuration server is needed to store their values).   When the CAS server is stopped then the cgroups are removed.   How to configure it? The administrator can assign users to these "priority level policies" based on their identity group memberships, or can explicitly assign priority level policies to individual users.   The way to do it is described in details in the SAS Viya 3.4 administration guide but basically the SAS Administrator has to define policies in the Consul store through the SAS Viya command line interface (sas-cli) and create corresponding Custom groups in Environment Manager to associate the end-users to the resource management groups.   Typically, you can define 1 to 5 "priority" groups and allocate quotas for global and session tables but also a share of the CPU capacity per priority group.   For example, you could create 3 priority groups: one for the VA users, one for the ETL users and one for the Data Scientists - with a CPU repartition of 20, 30, and 50% and define different quotas to limit the space used for global, session, or HDFS loaded tables for each group.   Why? It is a very interesting enhancement as you can limit the amount of space used in the CASlibs for your end users, but you can also avoid that a "crazy" data scientist, running a single but extremely heavy set of Machine-learning/advanced analytics actions takes all of the CPU power of your CAS cluster at the expense of the other end-user (maybe consuming simple VA reports or running Data processing jobs).   Overall, those options provide an even better control of the CAS server utilization and allow to divide the CAS resources between multiple groups of users of the same CAS environment.   Finally, there are other very interesting new resource management options such as cas.MAXCORES or env.CAS_ADDITIONAL_YARN_OPTIONS (for a better interaction with YARN).     Multi Tenancy improvements (different LDAP for each tenant)   SAS Viya 3.4 makes the Multi-tenancy model much more appealing because the very unique requirement on the LDAP structure is removed (1 Organization Unit (OU) and 1 Organization Unit called "provider).   With the new SAS Viya release, each tenant can have its own LDAP configuration and connection. Each tenant can have its users and groups in his own LDAP Directory or they can share the same one - but it is possible to define a specific LDAP connection query for each tenant. Another new feature of the Multi-tenancy is that you can choose, for the SAS Data Infrastructure Server, to use either different PostgreSQL schema or different PostgreSQL database for each tenant's user content.   Finally, with the capability to limit CPU consumption par CAS instances (cas.CPUSHARE), it is also easier to manage concurrent tenant's CAS servers sharing the same physical machine.   Conclusion   There are many other things with SAS Viya 3.4, such as new data integration capabilities (AWS S3, Spark, EMR support), improvement of the Compute and launcher servers (Kerberos support and launcher shared account). But this new release of SAS Viya brings great benefits : New features required by the business and the data science community Response to the field feedback and experience: answers to customers requests and fix for identified shortages in the features or additional Enterprise-ready capabilities. Thanks for reading! ... View more