What's new in SAS's Cloud Rapid Deployments (with a zoom on Azure)
With the recent updates of the Viya Quickstarts in the Cloud, I thought it would be a good time to come back on the Quickstart process and talk about some recent deployment experience in Azure.
SAS is a "Microsoft One" Commercial Partner which means that SAS and Microsoft teams closely collaborate on cloud technologies, solution areas and rapid deployment models, It makes Azure is a platform of choice for a SAS Viya deployment.
Also, with the move to Containers deployment in Kubernetes for the next Viya 4.0.1 release, we can expect the Cloud platforms such as Azure to become even more popular for Viya deployments.
So if you are an Architect or an Installation Engineer and completely new to Azure, now is probably a good time to get a first exposure to it 😊
The basics of the Viya Quickstart for Azure deployment are covered in the excellent post and 30 minute video from Michael Goddard. Note that the video is also available on YouTube, so any customer can use it to prepare a Quickstart deployment !
The purpose of this blog is to revisit the Quickstart deployment in Azure with a Viya 3.5 order and a CAS MPP environment but also to provide some extra guidance to secure your in Azure and make sure you don’t face some of the issues I encountered in my own deployment 😉
Before diving into the details, I'd like to thank Chris Lynch for his precious help in the write-up of this blog.
What’s new with the Viya Quickstarts
SAS's Quickstarts are getting better and better ! Not only have the three of them (AWS, GCP and Azure) have been updated to support the latest Viya version (3.5), but now, they also support CAS MPP configurations.
A new parameter has been added for that in each Quickstart and allows you to have up to 10 CAS workers in your Quickstart deployment. This makes Viya with CAS MPP deployments very easy.
As an example all you have to do is for the Azure Quickstart is to specify the number of CAS workers that you want, in the Quickstart template form (3 for example) :
Each Quickstart documentation (either README files on GitHub pages or PDF documentation for AWS), provides Architecture Diagrams for CAS SMP and CAS MPP cases.
See below the architecture diagram for the Quickstart architecture for SAS Viya on Azure with a CAS MPP configuration.
While the "Quick" word in "Quickstart" means "Quick deployment", it does not mean "Quick (and dirty) Architecture design".
To the contrary, one look a the architecture diagram is enough to see that many important Architecture principles and best practices for a Cloud deployment are implemented by the Quickstarts :
- Virtual network for Public/Private subnets separation
- Gateway/Load-balancer for Web applications access.
- Bastion Host for the administrative access
- Network Security Groups to restrict network access to specific ports and source machines
- Deployment Templates (declarative syntax and automation)
The Quickstarts are not simply provisioning a bunch of VMs to deploy our software. They also provision all the Cloud services or components required to create a robust and secured environment.
Of course, in a customized Cloud deployment, a more sophisticated topology and model could be implemented, but the Quickstart provides a solid starting point, that can be extended as the deployment templates are public.
So, with the addition of MPP CAS, it makes the Quickstart a valid option not only for demo or PoC/V (Proof of Concept/Value) but also for real project deployments for customers licensing a "Quickstart supported" solution (see below).
So let's recap on the changes for our Quick Start !
- Viya 3.5 support
- SAS Visual Analytics 8.5 on Linux
- SAS Visual Statistics 8.5 on Linux
- SAS Visual Data Mining and Machine Learning 8.5 on Linux
- SAS Data Preparation 2.5
- Viya-ARK 3.5 integrated
- Support of CAS MPP (up to 10 workers)
Regarding Azure, please note that there is also a new excellent post from Margaret Crevar (from the EEC team), with considerations on the Azure VM types, network and storage choices that impact the performances of our Software.
Tips and tricks for a Quickstart deployment in Azure
Ok, so now let's zoom on the Azure Quickstart and talk a little bit about the deployment.
As said previously, if you are fairly new to the Viya QuickStart on Azure, the best starting point is the Michael Goddard’s YouTube video : “SAS Viya Quickstart Template for Azure | Introduction and Demonstration”.
If you already know the basics and want to learn more advanced techniques like use Mirror repository or monitor the deployment progress, you will be interested by the following sections.
Use a mirror for your Azure Quickstart deployment
For many reasons, using a Mirror repository for the SAS packages is a best practice for any Viya deployment (see this post if if you need to remember why). Using a mirror also allows you to test the Quickstart with an Internal order.
According to the Quickstart README file, to use a Mirror repository, all you have to do is to:
- Upload the mirror content into an Azure Blob Storage ("Blob Storage" is the name of the storage service for Azure but each of the major Cloud provider has its own, S3 for AWS or Google Cloud Storage for GCP) with the Azure CLI,
- And provide the corresponding URI (for the mirror) as a parameter in the QuickStart template (before launching the deployment).
It seem very simple, right ? However, the Azure portal interfaces can sometimes be a little bit confusing, especially if you are new to them. So let's detail a little bit more the steps with a real example.
Upload the mirror
Let's start with the first step: "upoad all the mirror content"
- You need a an Azure Storage account. it is recommended to create it in the same region as the Quick Start Deployment resource group. But a good idea is to create the storage account for the mirror in a different resource group from the deployment’s one. Because, then you can remove the deployment by deleting the deployment resource group but maintain the mirror source storage in case you want to redeploy.
- In a storage account, you can store things in "blob containers", "file shares", "queues" or "tables", here we use the "blob containers".
- We can create a new container for the mirror called "mirrors" for example.
If you open the Storage explorer in the Azure portal, it should look like this:
- Now you need to find the access key corresponding to your Azure Storage account, you should find it in the properties of the storage account ("frarporgdiag" in this example).
You can copy the value of the Key field either for key1 or key2, that's what you will use for your "$STORAGEKEY" in the upload command that we need to run.
Now, you should be in position, to run the Azure CLI command to upload the mirror content into the blob container. The README file of the Quickstart gives the command line syntax :
az storage blob upload-batch --account-name "$STORAGE_ACCOUNT" --account-key "$STORAGEKEY" --destination "$SHARE_NAME" --destination-path "$SUBDIRECTORY_NAME" --source "$(pwd)"
- Connect to a machine that has access to the mirror repository source directory (for example located in /mirrordisk/09QSDS) and where the Azure CLI has been installed (for example with YUM) to run the “az” command.
Here is a real life example for the command using the previous example, values:
az storage blob upload-batch --account-name "frarporgdiag" --account-key "kKjO1f4evCuGZ1mC.....your key......UvE+pWsVmQ==" --destination "mirrors" --destination-path "09QSDS" --source "/mirrordisk/09QSDS"
From there, your screen should start to scroll pretty quickly as it reports each file upload into the Azure blob container 😊. As a result, a folder named "09QSDS" is created in your "mirrors" blog container and contains all the files and folder of the mirrored repository.
Actually, if you use the Azure CLI on linux and want to increase the automation of the process, there is even a trick do this all as one line without needing to get the key out of the web interface (by having bash fetch the key for you in a sub var):
az storage blob upload-batch --account-name "$STORAGE_ACCOUNT" --account-key "$(az storage account keys list --resource-group "$RESOURCE_GROUP" --account-name $STORAGE_ACCOUNT --subscription "$SUBSCRIPTION" --query "[0].value" | tr -d '"')" --destination "$SHARE_NAME" --destination-path "$SUBDIRECTORY_NAME" --source "$(pwd)"
Get the mirror URL
So far, so good. Now let's see the second part which is primarily to find the proper URL/URI to use to help our Quickstart deployment to find our brand new Mirror repository 😊
Here is the information that we have from the README file :
- Create a SAS key that has (at a minimum) List blob and Get blob privileges on the blob store.
- During deployment, set the DeploymentMirror parameter to the URL of the folder in the Azure blob that is qualified by that SAS key.
But don’t get confused ! in this context "SAS" means "Shared Access Signature" which is the Azure mechanism to provides secure delegated access to resources in your storage account (such as the "mirrors" blob containers) without compromising the security of your data. See this page for additional details.
Here are the steps from the portal to create the "SAS" key:
- In your storage group, select "Shared access signature" on the left side menu.
Now create a new one with:
- allowed services
– blob
- allowed resource types- All of them
- allowed permissions - Read List
And set the end date far enough in the future to cover your install window.
Then click "generate SAS and Connection String" and grab the value in the "SAS Token" string
If like me, you prefer to use the new “Storage explorer” feature, it would look like this:
Right click on "Mirrors"
Select "Get Shared Access Signature".
Set the expiry time (the "SAS" key used by the Quickstart to download the packages from the mirror repository will only be valid for this period of time), For the permissions, check "Read" and "List". Then click on the "Create" button, to get your URI, that contains the "SAS" key.
We are now in position to build the URL to set in the Viya Quickstart template form ! But before using it, we want to make sure the URL/URI actually works !
So copy the URI that is shown in the portal screen above, for example :
https://frarporgdiag.blob.core.windows.net/mirrors?sp=rl&st=2020-03-20T15:19:05Z&se=2020-03-21T15:19:05Z&sv=2019-02-02&sr=c&sig=WNlrg2w8zO5Ji8jh5KfVvPq379eLIV7OyyoO%2FKgdrcA%3D
If we use this URL directly, it does not work:
We need to manipulate a little bit the url to get the form that is required for the Quickstart template.
Keep the first part of URI string (until the "?"), but then modify the path, as if you wanted to point on a specific file that is available in your mirror repository, for example : entitlement.json.
In our example that would be :
https://frarporgdiag.blob.core.windows.net/mirrors/09QSDS/entitlements.json
Then add the Query string (starting with the "?" that contains the validity dates and the signature/ token)
https://frarporgdiag.blob.core.windows.net/mirrors/09QSDS/entitlements.json?sp=rl&st=2020-03-20T15:19:05Z&se=2020-03-21T15:19:05Z&sv=2019-02-02&sr=c&sig=WNlrg2w8zO5Ji8jh5KfVvPq379eLIV7OyyoO%2FKgdrcA%3D
Now, it is working :
Now, it's time to set the value for the "Deployment Mirror" parameter in the QuickStart template.
So to get the final URL, we just need to "chop off" the entitlements.json, it gives something like that :
https://frarporgdiag.blob.core.windows.net/mirrors/09QSDS/?sp=rl&st=2020-03-20T15:19:05Z&se=2020-03-21T15:19:05Z&sv=2019-02-02&sr=c&sig=WNlrg2w8zO5Ji8jh5KfVvPq379eLIV7OyyoO%2FKgdrcA%3D
It won't work in a web browser, but ,trust me, this is what needs to be set in the Quickstart template.
Monitor and troubleshoot your deployment
As you have seen, the value for some of the template parameters can be a little bit tricky to find. So you surely want to monitor your deployment as closely as possible to make sure everything works as expected. Shortly after clicking on the "Purchase" button that launch the Quickstart deployment, the Ansible Controller Virtual machine is created and you can connect to it with the "vmuser" account. As soon as you see in the Azure portal that the "Ansible" Virtual Machine has been created, get the public IP address and connect to it from your ssh client.
You need to provide the SSH private key corresponding that corresponds to the SSH public key that was set in the Quick Start template. From there, you can monitor the deployment log exactly the same way you would to it from an on-premise deployment. For example:
[centos@frarpo-jumphost ~]$ ssh -i ~/.ssh/frarpo-viyadep-key vmuser@52.168.39.103 [vmuser@Ansible]$ tail -f /var/log/sas/install/viya_deployment.log
At the end, you can also check the Viya services status using the Viya ARK MMSU scripts (embedded in the deployment):
[vmuser@Ansible ~]$ cd /sas/install/ansible/sas_viya_playbook [vmuser@Ansible sas_viya_playbook]$ ansible-playbook viya-ark/playbooks/viya-mmsu/viya-services-status.yml
My deployment has failed…or maybe not?
On my first attempt, quite optimistically, I decided to deploy a Viya environment with 3 CAS workers and to keep the default Azure instance types, predefined in the Quickstart template:
But near the end of the deployment I had the unpleasant surprise to see messages below in the Azure Portal for my running deployment :
Humm...It does not look good, does it ? However, using the described technique above I was closely monitoring the deployment via the ansible deployment log and noticed that the deployment log shows no failure for the ansible deployment. Moreover, when I checked the status of my Viya services they all reported an "up and running" status. I even can logon with success to SAS Environment manager where everything seems perfect (with my 3 CAS workers as requested):
So...is my deployment valid or not ?
With a better look at the error message I can isolate an interesting message:
"After restart, services were still found to have not successfully started. This likely indicates that the boxes have been undersized. Try increasing the size of the services box."
Thanks to the Development team, I had an explanation:
"During the install process, the startup of the Viya services becomes very cpu bound. Once everything loads into ram, they settle down a lot, but starting all the services tends to be the point where a larger box is needed for a bit. But the services definitely can come online after the timeouts for the build".
Unfortunately some Azure time out settings (that can not be moved past) introduce a time limit for some tasks that the deployment process cannot meet because of limited CPU resources.
So, although my Viya deployment was successful, my Azure deployment was not.
Of course it would have been better to have a completed successful Azure deployment but as long as there is no failure in the ansible deployment results and all the Viya services are up and running, we can use Viya as is or increase the instances sizes and redeploy in case we want more reinsurance on how it performs.
Azure deployments are more records of what happened (we can redeploy or undeploy a deployment, but resource groups are really the management object that azure uses).
As the Quickstart has its own resource group and is expected to be the only thing in that resource group so it can also be cleaned up as a unit (also prevents name collisions).
Conclusion
Hopefully after reading this article you know a little bit more about the SAS's Quickstarts (and learned from my own experience and mistakes 😊 ). The Quickstarts are a great way to automate your deployments of Viya in the Public Cloud infrastructure while complying with the acknowledged rules of the art of Cloud deployments.
