Yes, you can create a Microsoft Azure Resource Manager template for SAS Viya products

[#1]

This post take you through the high level steps and some of the catches with Azure to create an end-to-end SAS Viya product deployment. Keep in mind this is a simple deployment, not focused on creating a production environment. This environment does not include the required security considerations necessary to protect your environment and your assets. In addition, prior to deploying SAS Viya, an infrastructure-sizing exercise should be performed so your environment meets your performance expectations.

To go from zero to running, we have to look at SAS Viya product installation and configuration capabilities. If this is not easily automatable, then we would be stuck.  SAS designed the SAS Viya products to use Ansible, an open-source automation engine, which installs and configures the SAS Viya product on one or more virtual or physical servers.  Now that we can automate the software deployment, what about the cloud deployment?

Microsoft Azure has the Azure Resource Manager capability which allows you to create all the resources you need and to execute OS commands to orchestrate an end-to-end deployment. The Azure Resource Manager template is a json file that maps out all the resources needed. It executes all this with a single command.  You can learn more about how to create your first simple Azure Resource Manager template.

With SAS using Ansible and Azure Resource Manager template, I can easily automate the deployment of a SMP and MPP SAS Viya product from zero to running with a few simple steps. Below are the detailed steps to automate SAS Visual Data Mining and Machine Learning (VDMML) in Azure. I started with no VM(s) and about 45 mins later I had it running and ready to use VDMML Visual system.

Step 1 - Pre-Steps

These are the items that need to be done to allow for end-to-end automation of the SAS Viya product in Azure cloud, allowing the resulting deployment to be a system ready for users to access.

1. SAS Software Order Email (SOE) – A SAS Viya product order from SAS. Please contact your sales rep.
2. LDAP – SAS Viya products require an LDAP for the Visual interfaces. You will need to add your LDAP configuration information to the SAS Viya deployment configuration script, sitedefault.yml, so the deployment will configure the software to your LDAP.
3. SAS deployment script – Script that tweaks the OS from missing pre-requirements, installs Ansible, and runs the SAS Viya product Ansible playbook.  This is simply automating the few commands noted in the SAS Viya deployment guide.  This would automate the SSSD configuration with the LDAP on the OS (nsswitch and pam) for SAS Studio security, which is optional.  Please talk with SAS if you would like to see an example.

Some people might be asking, what about the SAS Software Depot? With SAS Viya products, we moved this to a RPM repository that is on the internet.  Now, Azure instances can go directly to SAS to pull down the specific packages needed versus setting up a special local SAS Software Depot.  Note, you can create a mirror RPM repository, similar to other repositories like RedHat, and point the deployment to use the mirror (please refer to the SAS Deployment Guide for more details on how to create a mirror repository).

Azure Cloud Automation Steps

To automate the Viya product deployment in Azure, I used the following Azure capabilities:

1. Azure Storage Account – files storage that can be used for a deployment
2. Azure Resource Manager template – json to define one or more resources to deploy to a resource group
3. Azure Cloud Shell – command line interface to the Azure CLI

Azure Storage Account

Here we can store files used by the Viya Azure deployment process. The Azure Storage Account has two resource locations for files:

• Blobs – files that can be copied into the instances
• Files – files that can be used by the Azure Cloud Shell

Azure Storage Account Blobs 

Inside the “Blobs” area of the Storage Account, I created a “Private (no anonymous access)” container to place my helper scripts. You could choose to allow this container to be exposed to anonymous read access if you want, but this is exposed to the internet and you don’t want people to take your license file.

Here I placed the following files in the “Blobs”:

1. SAS Ansible Playbook (SAS_Viya_playbook.tgz) - SAS Playbook file from the SOE
2. LDAP configuration file (sitedefault.yml) - LDAP properties for the SAS Viya environment to use, configured by the SAS playbook.  Note, it is recommend to add the administrator property so you don't need to use "sasboot".
3. SAS deployment script (e.g. SASDeploy.sh) – Custom script you create to tweak the OS, install Ansible, and run the SAS playbook.  Basically a simple cut and paste command from the SAS Deployment Guide for SAS Viya product.
4. azureMPPSetup.sh (recommended for SAS Viya MPP) – SAS Viya MPP deployment for Azure cloud is needed to tweak the primary VM instance and run the SAS Deployment script to simplify the command in Azure Deployment template custom script command
5. id_rsa (recommended for SAS Viya MPP) – private ssh key to passwordless-ssh between VM instances, this is needed for Ansible to run as the OS admin user without providing password (Azure CLI accesses the VM instance as root which doesn’t have passwordless ssh to another VM instance)

Azure Storage Account Files

Inside the “Files” area of the Storage Account, I create a folder for all the Resource Manager templates (json files):

Azure Resource Manager template

I learned that creating an Azure Resource Manager template requires a lot of trial and error. The documentation can be confusing on what parameters values to set and which are required versus optional.  The deployment failures do provide more details on the required parameter.  After some work, I gained an understanding of how to deploy in Azure and create a working template.  Note, someone who is an Azure expert would know better places to obtain details for creating an Azure Resource Manager template.

With the Azure Resource Manager template, I did find some interesting things:

1. Global properties – can be set either by the resource manager deployment user (parameters) or hardcoded in template (variables). The parameters do allow for a more interactive, customized deployment experience with the Resource Manager deployment user, but not everything needs to be exposed helping to simplify the deployment.
2. Microsoft.Compute/virtualMachines -> “osProfile” – here we want to set the “adminUsername” and “publicKeys” so you can log into the VM and this will be the public key for the MPP private id_rsa key, so the Ansible script can passwordless-ssh to all vms during the deployment.
3. Microsoft.Azure.Extensions customscript – this is how you can copy files into the deployment and execute an OS command. Note, you can copy as many files are you want from the Azure Storage Account Blobs (because I secure that container, you need to add “storageAccountName” and “storageAccountKey”) or any public url.  This is also where you can execute a single OS command string (if you want to execute multiple OS commands, you need to use “;” between commands).  This is where you can create a script (azureMPPSetup.sh) and simplify the single command.  Azure Resource Manager templates have the customScript json object inside the “resources” section of the specific “virtualMachine”.  This customScript object also needs to “dependsOn” that specific “virtualMachine”.  That is a little weird to reference an object when this has to be inside the object, but it is required.
4. Microsoft.Network/networkSecurityGroups – need to expose both the https port and ssh port in “securityRules” section (default is ssh port, but if you specify a securityRule for the https port, then you need to add back the ssh port or else you won’t be able to ssh into this VM instance).
5. MPP Deployment Only: Microsoft.Compute/virtualMachines dependsOn – needs to have a reference to the worker virtual machine loop so this does not execute (or you can place this dependency on the customScript) until all the worker VMs are created and ready.
6. MPP Deployment Only: PublicIpAddress and networkSecurityGroups – only needs to be created for the primary VM where the Microservices, Web Applications and CAS controller are deployed. All other worker VMs will only have internal ips and be exposed on the internal Microsoft network.  This helps reduce the exposure to the internet.
7. MPP Deployment Only: worker vm customScript – this is only needed to tweak the default OS linking the /opt/sas directory to the large mounted storage location.

Azure Cloud Shell

Here we have access to the Azure CLI to execute the deployment of an Azure Resource Manager template. When you start a Cloud Shell, you can see the Storage Account Files:

Now we can execute the Resource Manager deployment command to create our solution:

az group deployment create --resource-group GTP --template-file clouddrive/templates/<name>

Note, you can add the “-no-wait” option to the command so the Azure Cloud Shell returns to do other commands. Then you can monitor the deployment by watching as the various resources are created and then watch the “Extensions” inside the primary virtual machine.  Here you can see the custom script status.  You will see the status of “Provisioning succeeded” when it completes and then your end-to-end environment is ready to use. When it is still running the scripts in the instance, the status is “Transitioning”.

Interesting facts about Microsoft Azure cloud:

1. Azure Resource Manager template only supports a single custom script extension (you can use the older “Microsoft.OSTCExtensions” and newer “Microsoft.Azure.Extensions” to have multiple, but that did not really help much)
2. Azure Resource Manager template is a json file, so you can’t format long lines nicely with returns and concatenating strings for variables can be more complex
3. Azure creates separate resources when creating a virtual machine, so deleting requires you to clean up all the pieces (virtual machine, Disk, Network interface, Public IP address, Network security group)
4. Azure requires unique instances of resource so you can’t share one common Network security group for all virtual machines
5. Azure Cloud Shell random has expired token failures, which require you to just exit and restart Cloud Shell. Example error:
   1.  james@Azure:~$ az group deployment create –resource-group GTP –template-file clouddrive/templates/azureMPP.json
   2.  Deployment failed. Deployment failed. The access token expiry UTC time ‘9/28/2017 6:19:42 PM’ is earlier than current UTC time ‘9/28/2017 6:20:05 PM’.

      6. Azure cloud mounts the extra disk as unique directory (so not added to the root partition), so you need to softlink or move                  items to the specific directory.

And that’s it

This post shows how a simple VDMML Visual environment can stand up quickly in Azure for a play environment. The SMP deployment took around 34 minutes and the MPP took around 46 minutes.  This is not focused on the production environment where a client will have more security wrapped around and a sizing done.  But this shows two things:

1. SAS Viya products can run in Microsoft Azure cloud
2. Azure Resource Manager template and SAS usage of Ansible can automate the end-to-end deployment

Resulting in: