SAS software always does a great job allowing clients to update the SAS level and know their SAS code will still execute.  With Viya and SAS 9.4, we also made sure the Hadoop In-DB libraries (EP) were common allowing each software level to execute with the same capabilities (scoring and parallel data movement).  What I found is the SAS In-DB published score codes differ between versions (which is fine) and requires the score execution method to be modified to work with the different published score codes.  Note, even with SPRE engine in a Viya deployment, the SAS 9.4 scoring macros are disabled in Viya SPRE engine. What you need to know about Publishing Score Code in Hadoop: SAS publishes two types of score files in Hadoop for In-DB scoring: 1)     DS2 (.ds2 file) – raw DS2 score code that can prepare and score the data 2)     Item Store (.is file) – binary score code that can contain ds2 (prepare and maybe score) and the Analytic Store (astore - binary score file) When you have a SAS data step score code, the publishing process in SAS 9.4 and Viya will translate the data step code to DS2 and then publishes that into Hadoop. SAS 9.4 can publish score code as DS2 or as Item Store in Hadoop based on the score code type.  SAS 9.4 publishes majority of the score code as DS2 score code unless you have an astore.  Only HPFOREST, HPSVM and SAS Function Compiler (FCMP) can create an astore file in SAS 9.4 which means these are the only models in SAS 9.4 that will publish an Item Store in Hadoop. SAS Viya publishes all score code (data step, DS2 and astore) as an Item Store in Hadoop. In the “Detail Code examples” sections, you can see the details on how to publish score code for each SAS level.   What you need to know about Executing Score code in Hadoop: Both SAS 9.4 and Viya use the same In-DB binaries in Hadoop to run score code which means either SAS level can execute published score code from any version.  The only catch is the SAS level method to execute the different published score code files (.ds2 or .is). In the “Detail Code examples” sections, you can see how to execute score code for each SAS level.  Here are some high-level comments on executing score code in each version.   SAS 9.4 Execute model: When you execute a published model using the run model macro on SAS 9.4, you can specify either a DS2 file or an Item Store file stored in Hadoop: 1.            To run a model stored as DS2 program in Hadoop, you use the scorepgm= option. 2.            To run a model stored as an Item Store om Hadoop, you use the store= option. Make sure to check Hadoop’s published score code directory for the type of file (.ds2 or .is) to select the correct option when executing score code in SAS 9.4.  Remember, if the model was published by Viya, you use the “store” option in the macro because they are published as Item Store.   Viya Execute model: Remember, the SPRE engine in Viya is similar to SAS 9.4, but the scoring macros are disabled in Viya.  When you execute a published model using the SPRE proc in Viya (PROC SCOREACCEL), this can ONLY execute Item Store publish score codes.  This works great for all Viya models and only SAS 9.4 astore published models (very limited list with SAS 9.4 - HPFOREST, HPSVM and FCMP).  Note, this SPRE proc will convert to call CAS Action set to score the model. When you execute a published model using the CAS action set (modelPublishing) in Viya, you can choose to execute an Item Store or DS2 published score code.  This is the only way in Viya to execute all SAS 9.4 published models and Viya published models in Hadoop.   Detail Code examples Below are examples on publishing and executing score code in Hadoop for each software level.   Viya Publishing Models (using SPRE vs CAS action): With Viya, you can choose to use the SPRE procedure or CAS action.  Here is an example of using SPRE procedure to publish score code: proc scoreaccel;    publishmodel       target=hadoop       caslib="caslibhadoop"       modelname="&scorename"       modeltype=DS       filelocation=local       programfile="&scoredir./&scorename..sas"       xmlfile="&scoredir./&scorename..xml"       outdir="&scoredir./out"       modeldir="/sasmodels"       classpath="/hadoop/lib:/hadoop/conf"       ; quit;     Viya Execute Score code Here is the SPRE procedure to execute publish score code, but this can ONLY execute Item Store published score code (all Viya models and only SAS 9.4 astore models): proc scoreaccel;    runmodel       target=hadoop       caslib="caslibhadoop"       modelname="&scorename"       modeldir="/sasmodels"       server=" hadoophostname "       intable="datatoscore"       outtable="datascored"       forceoverwrite=yes       ; quit;   Here is the CAS action that can execute any model published in Hadoop (Item Store AND DS2), this is the only way for Viya to execute all of the SAS 9.4 models published: proc cas;   loadactionset "modelPublishing";   action runModelExternal submit /       modelName="&scorename"       externalOptions={           extType="hadoop",           server="hadoophostname",           modeldir="/&scorename",           classpath="/hadoop/lib:/hadoop/conf",           ds2file="/sasmodels/&scorename./&scorename..ds2",           inTable=" datatoscore",           outTable=" datascored ",           forceOverwrite=TRUE,           trace=TRUE       }           ; run;       SAS 9.4 Publish Scoring: Here is the SAS 9.4 publish macro to publish DS2 score code in Hadoop: %INDHD_PUBLISH_MODEL(dir=&scoredir., datastep=&scorename..sas, xml=&scorename..xml,                       modeldir=/sasmodels, modelname=&scorename., action=replace); Note, this assumes you have both the data step code and xml file locally so the macro can convert this to DS2 code in Hadoop.   Here is the SAS 9.4 publish macro to publish Item Store score code in Hadoop: %INDHD_PUBLISH_MODEL(dir=&scoredir., store=&scorename..sasast, datastep=&scorename..sas,                       modeldir=/sasmodels, modelname=&scorename., action=replace); Note, this assume you have the analytic store and sas code to run that astore so it can convert to an Item Store in Hadoop.  Here is example code to generate the sas code for astore: proc astore;      describe store="&scoredir./&scorename..sasast" epcode="&scoredir./&scorename..sas"; run; Note, this macro is ONLY usable in SAS 9.4.  The SPRE engine in Viya does not have this macro, so you need to use the Viya proc/CAS action.   SAS 9.4 Score: Here is the SAS 9.4 execute macro to execute DS2 score code in Hadoop: %INDHD_RUN_MODEL(inputtable=hdplib.datatoscore, outputtable=datascored,                scorepgm=/sasmodels/&scorename./&scorename..ds2, forceoverwrite=true);   Here is the SAS 9.4 execute macro to execute Item Store score code in Hadoop: %INDHD_RUN_MODEL(inputtable=hdplib.datatoscore, outputtable=datascored,                store=/sasmodels/&scorename./&scorename..is, forceoverwrite=true);   Note, this macro is ONLY usable in SAS 9.4.  The SPRE engine in Viya does not have this macro, so you need to use the Viya proc/CAS action.   ... View more

  This post take you through the high level steps and some of the catches with Azure to create an end-to-end SAS Viya product deployment. Keep in mind this is a simple deployment, not focused on creating a production environment. This environment does not include the required security considerations necessary to protect your environment and your assets. In addition, prior to deploying SAS Viya, an infrastructure-sizing exercise should be performed so your environment meets your performance expectations.   To go from zero to running, we have to look at SAS Viya product installation and configuration capabilities. If this is not easily automatable, then we would be stuck.  SAS designed the SAS Viya products to use Ansible, an open-source automation engine, which installs and configures the SAS Viya product on one or more virtual or physical servers.  Now that we can automate the software deployment, what about the cloud deployment?   Microsoft Azure has the Azure Resource Manager capability which allows you to create all the resources you need and to execute OS commands to orchestrate an end-to-end deployment. The Azure Resource Manager template is a json file that maps out all the resources needed. It executes all this with a single command.  You can learn more about how to create your first simple Azure Resource Manager template.   With SAS using Ansible and Azure Resource Manager template, I can easily automate the deployment of a SMP and MPP SAS Viya product from zero to running with a few simple steps. Below are the detailed steps to automate SAS Visual Data Mining and Machine Learning (VDMML) in Azure. I started with no VM(s) and about 45 mins later I had it running and ready to use VDMML Visual system.   Step 1 - Pre-Steps These are the items that need to be done to allow for end-to-end automation of the SAS Viya product in Azure cloud, allowing the resulting deployment to be a system ready for users to access. SAS Software Order Email (SOE) – A SAS Viya product order from SAS. Please contact your sales rep. LDAP – SAS Viya products require an LDAP for the Visual interfaces. You will need to add your LDAP configuration information to the SAS Viya deployment configuration script, sitedefault.yml, so the deployment will configure the software to your LDAP. SAS deployment script – Script that tweaks the OS from missing pre-requirements, installs Ansible, and runs the SAS Viya product Ansible playbook.  This is simply automating the few commands noted in the SAS Viya deployment guide.  This would automate the SSSD configuration with the LDAP on the OS (nsswitch and pam) for SAS Studio security, which is optional.  Please talk with SAS if you would like to see an example. Some people might be asking, what about the SAS Software Depot? With SAS Viya products, we moved this to a RPM repository that is on the internet.  Now, Azure instances can go directly to SAS to pull down the specific packages needed versus setting up a special local SAS Software Depot.  Note, you can create a mirror RPM repository, similar to other repositories like RedHat, and point the deployment to use the mirror (please refer to the SAS Deployment Guide for more details on how to create a mirror repository).   Azure Cloud Automation Steps To automate the Viya product deployment in Azure, I used the following Azure capabilities: Azure Storage Account – files storage that can be used for a deployment Azure Resource Manager template – json to define one or more resources to deploy to a resource group Azure Cloud Shell – command line interface to the Azure CLI   Azure Storage Account Here we can store files used by the Viya Azure deployment process. The Azure Storage Account has two resource locations for files: Blobs – files that can be copied into the instances Files – files that can be used by the Azure Cloud Shell   Azure Storage Account Blobs  Inside the “Blobs” area of the Storage Account, I created a “Private (no anonymous access)” container to place my helper scripts. You could choose to allow this container to be exposed to anonymous read access if you want, but this is exposed to the internet and you don’t want people to take your license file.   Here I placed the following files in the “Blobs”: SAS Ansible Playbook (SAS_Viya_playbook.tgz) - SAS Playbook file from the SOE LDAP configuration file (sitedefault.yml) - LDAP properties for the SAS Viya environment to use, configured by the SAS playbook.  Note, it is recommend to add the administrator property so you don't need to use "sasboot". SAS deployment script (e.g. SASDeploy.sh) – Custom script you create to tweak the OS, install Ansible, and run the SAS playbook.  Basically a simple cut and paste command from the SAS Deployment Guide for SAS Viya product. azureMPPSetup.sh (recommended for SAS Viya MPP) – SAS Viya MPP deployment for Azure cloud is needed to tweak the primary VM instance and run the SAS Deployment script to simplify the command in Azure Deployment template custom script command id_rsa (recommended for SAS Viya MPP) – private ssh key to passwordless-ssh between VM instances, this is needed for Ansible to run as the OS admin user without providing password (Azure CLI accesses the VM instance as root which doesn’t have passwordless ssh to another VM instance)   Azure Storage Account Files Inside the “Files” area of the Storage Account, I create a folder for all the Resource Manager templates (json files):       Azure Resource Manager template I learned that creating an Azure Resource Manager template requires a lot of trial and error. The documentation can be confusing on what parameters values to set and which are required versus optional.  The deployment failures do provide more details on the required parameter.  After some work, I gained an understanding of how to deploy in Azure and create a working template.  Note, someone who is an Azure expert would know better places to obtain details for creating an Azure Resource Manager template. With the Azure Resource Manager template, I did find some interesting things: Global properties – can be set either by the resource manager deployment user (parameters) or hardcoded in template (variables). The parameters do allow for a more interactive, customized deployment experience with the Resource Manager deployment user, but not everything needs to be exposed helping to simplify the deployment. Microsoft.Compute/virtualMachines -> “osProfile” – here we want to set the “adminUsername” and “publicKeys” so you can log into the VM and this will be the public key for the MPP private id_rsa key, so the Ansible script can passwordless-ssh to all vms during the deployment. Microsoft.Azure.Extensions customscript – this is how you can copy files into the deployment and execute an OS command. Note, you can copy as many files are you want from the Azure Storage Account Blobs (because I secure that container, you need to add “storageAccountName” and “storageAccountKey”) or any public url.  This is also where you can execute a single OS command string (if you want to execute multiple OS commands, you need to use “;” between commands).  This is where you can create a script (azureMPPSetup.sh) and simplify the single command.  Azure Resource Manager templates have the customScript json object inside the “resources” section of the specific “virtualMachine”.  This customScript object also needs to “dependsOn” that specific “virtualMachine”.  That is a little weird to reference an object when this has to be inside the object, but it is required. Microsoft.Network/networkSecurityGroups – need to expose both the https port and ssh port in “securityRules” section (default is ssh port, but if you specify a securityRule for the https port, then you need to add back the ssh port or else you won’t be able to ssh into this VM instance). MPP Deployment Only: Microsoft.Compute/virtualMachines dependsOn – needs to have a reference to the worker virtual machine loop so this does not execute (or you can place this dependency on the customScript) until all the worker VMs are created and ready. MPP Deployment Only: PublicIpAddress and networkSecurityGroups – only needs to be created for the primary VM where the Microservices, Web Applications and CAS controller are deployed. All other worker VMs will only have internal ips and be exposed on the internal Microsoft network.  This helps reduce the exposure to the internet. MPP Deployment Only: worker vm customScript – this is only needed to tweak the default OS linking the /opt/sas directory to the large mounted storage location.   Azure Cloud Shell Here we have access to the Azure CLI to execute the deployment of an Azure Resource Manager template. When you start a Cloud Shell, you can see the Storage Account Files:         Now we can execute the Resource Manager deployment command to create our solution: az group deployment create --resource-group GTP --template-file clouddrive/templates/<name>     Note, you can add the “-no-wait” option to the command so the Azure Cloud Shell returns to do other commands. Then you can monitor the deployment by watching as the various resources are created and then watch the “Extensions” inside the primary virtual machine.  Here you can see the custom script status.  You will see the status of “Provisioning succeeded” when it completes and then your end-to-end environment is ready to use. When it is still running the scripts in the instance, the status is “Transitioning”.   Interesting facts about Microsoft Azure cloud: Azure Resource Manager template only supports a single custom script extension (you can use the older “Microsoft.OSTCExtensions” and newer “Microsoft.Azure.Extensions” to have multiple, but that did not really help much) Azure Resource Manager template is a json file, so you can’t format long lines nicely with returns and concatenating strings for variables can be more complex Azure creates separate resources when creating a virtual machine, so deleting requires you to clean up all the pieces (virtual machine, Disk, Network interface, Public IP address, Network security group) Azure requires unique instances of resource so you can’t share one common Network security group for all virtual machines Azure Cloud Shell random has expired token failures, which require you to just exit and restart Cloud Shell. Example error:  james@Azure:~$ az group deployment create –resource-group GTP –template-file clouddrive/templates/azureMPP.json  Deployment failed. Deployment failed. The access token expiry UTC time ‘9/28/2017 6:19:42 PM’ is earlier than current UTC time ‘9/28/2017 6:20:05 PM’.       6. Azure cloud mounts the extra disk as unique directory (so not added to the root partition), so you need to softlink or move                  items to the specific directory.   And that’s it This post shows how a simple VDMML Visual environment can stand up quickly in Azure for a play environment. The SMP deployment took around 34 minutes and the MPP took around 46 minutes.  This is not focused on the production environment where a client will have more security wrapped around and a sizing done.  But this shows two things: SAS Viya products can run in Microsoft Azure cloud Azure Resource Manager template and SAS usage of Ansible can automate the end-to-end deployment Resulting in:       ... View more

I hear that question and wonder why people would ask. SAS Viya products can be installed on a virtual machine (VM) that meets the software requirements.  Google Cloud Platform (GCP) has an Infrastructure as a Service (IaaS) capability, Google Compute Engine (GCE), which can create a VM instance with the required OS and resources for SAS Viya products.  So what is the issue?   But the question is really “how can GCP make the SAS Viya product deployments easier” to allow me to go from zero to running the SAS solution in GCP.   This post takes you through the steps on how to achieve this goal. Keep in mind this is a simple deployment, not focused on creating a production environment.  This environment does not include the required security considerations necessary to protect your environment and your assets.  In addition, prior to deploying SAS Viya product, an infrastructure-sizing exercise should be performed so your environment meets your performance expectations.   To be able to go from zero to running, we have to look at SAS Viya products’ install and configuration capabilities. If this is not easily automatable, then we would be stuck.  SAS designed the SAS Viya products to use Ansible, open-source automation engine, which installs and configures the SAS Viya product on one or more operating systems.  Now that we can automate the software deployment, what about the cloud deployment?   With Google, GCP has a Cloud Deployment Manager capability which allows you to create a template of the whole deployment so you can use one command to standup everything. But I did something even simpler with GCP and used the Google Cloud Shell with the Google Cloud APIs.  The Google Cloud APIs provide simple commands to create GCP objects and execute commands inside GCE instances.   With SAS using Ansible and Google Cloud APIs, I can easily automate the deployment of a SMP and MPP SAS Viya product from zero to running with a few simple steps. Below are the detailed steps to automate SAS Visual Data Mining and Machine Learning (VDMML) in Google.   I started with no VM(s) and about 30 mins later I had a running ready to use VDMML Visual system.   Step 1 - Pre-Steps These are items that need to be done to allow for end-to-end automation of the SAS Viya product in Google Cloud allowing the resulting deployment be a system ready for users to access. Software Order Email (SOE) – A SAS Viya product order from SAS.  Please contact your sales rep. LDAP – SAS Viya products require an LDAP for the Visual interfaces. This is documenting your LDAP configuration information in a sitedefault.yml file, SAS Viya deployment configuration script, so the deployment will configure the software to the LDAP. SAS deployment script – Script that tweaks the OS from missing pre-requirements, install Ansible, and run the SAS Viya product Ansible playbook.  This is simply automating the few commands noted in the SAS Viya deployment guide.  This would automate the SSSD configuration with the LDAP on the OS (nsswitch and pam) for SAS Studio security, which is optional.  Please talk with SAS if you would like to see an example. Some SAS people might be asking, what about the SAS Software Depot? With SAS Viya products, we moved this to a RPM repository that is on the internet.  Now, GCE instances can go directly to SAS to pull down the specific packages needed versus setting up a special local SAS Software Depot.  Note, you can create a mirror RPM repository, similar to other repositories like RedHat, and point the deployment to use the mirror (please refer to the SAS Deployment Guide for more details on how to create a mirror repository).   Here is a screen shot of my GCP project where I just created one instance for my LDAP:   Step 2 – Storing Files in Google Cloud Platform Storage To automate the deployment in Google Cloud, we use the GCP Storage bucket for all the files the automation pushes into the GCP Compute Engine instances. The user would create a Google Cloud Platform Storage bucket and place the following files: SAS Ansible Playbook (SAS_Viya_playbook.tgz) - SAS Playbook file from the SOE LDAP configuration file (sitedefault.yml) - LDAP properties for the SAS Viya environment to use and configured by the SAS playbook.  Note, it is recommend to add the administrator property so you don't need to use "sasboot". SAS deployment script (e.g. SASDeploy.sh) – Custom script you create to tweak the OS, install Ansible, run the SAS playbook.  Basically a simple cut and paste commands from the SAS Deployment Guide for SAS Viya product. Here is a screen shot of my GCP Storage bucket with the files:   Step 3 - Google Cloud API commands In the Google Cloud Shell, you will access the Google Cloud SDK for Cloud APIs or you can install the Google Cloud SDK on your own system. The following commands are how you can deploy a SMP or MPP VDMML environment with Google Cloud APIs: # Create instance and wait to running gcloud compute instances create "${instanceName}" --zone "us-east1-c" \ --machine-type "n1-standard-32" --subnet "default" --tags "https-server" \ --image "rhel-6-v20170829" --image-project "rhel-cloud" --boot-disk-size "200"\ --boot-disk-type "pd-ssd" --boot-disk-device-name "${instanceName}" # Move files from Google Storage bucket into the GCE instances # Copy the SAS playbook and uncompress the file so it is ready to be use: gcloud compute ssh ${instanceName} --zone us-east1-c --command \ "gsutil cp gs://${gstorageLoc}/SAS_Viya_playbook.tgz ~/SAS_Viya_playbook.tgz;tar xzvf ~/SAS_Viya_playbook.tgz" # Copy the LDAP configuration file to the appropriate directory gcloud compute ssh ${instanceName} --zone us-east1-c --command \ "gsutil cp gs://${gstorageLoc}/sitedefault.yml ~/sas_viya_playbook/roles/consul/files/sitedefault.yml" # Copy the SAS deployment script and make it executable on the OS gcloud compute ssh ${instanceName} --zone us-east1-c --command \ "gsutil cp gs://${gstorageLoc}/SASDeploy.sh ~/SASDeploy.sh;chmod 777 ~/SASDeploy.sh" # Execute the SAS deployment script gcloud compute ssh ${instanceName} --zone us-east1-c --command "~/SASDeploy.sh"   Resulting deployment in GCP:      Note, some tweaks for MPP deployment: You would need to call “gcloud compute instances create” multiple times to create the instances. This can easily be done in a loop, just make sure to remember the instance names for the SAS deployment script. Use the Google Cloud API user for the ansible multi-machine install, so we need to setup passwordless ssh for that user by copying the private key to the main instance. (Optional) Recommended to only enable "https" port exposed to the internet on the SAS Viya management node. The SAS Viya workers don't need to expose any more ports to internet.   And that’s it This post shows how a simple VDMML Visual environment can stand up quickly in GCP for a play environment. The SMP deployment took around 22 mins and the MPP took around 30 mins.  This is not focused on the production environment where a client will have more security wrapped around and a sizing done.  But this shows two things: SAS Viya products can run in Google Cloud Platform Automation is easy with Google Cloud APIs and SAS usage of Ansible ... View more