Content Assessment is a collection of applications designed to help you understand various characteristics of your SAS 9.4 deployment. SAS Content Assessment is being updated every month to match the Viya development efforts in the area of migration from SAS 9.4 to Viya. Recent development has added new applications to Content Assessment which extend its capabilities beyond the assessment of a SAS 9.4 environment and begin to assist with the execution of migration to SAS Viya. In this post, I will look at these new applications     Content Assessment is available as a download from support.sas.com. The download contains the installation and usage documentation which is also available in the Migration section of the SAS Viya Administration guide.   The following resources provide more information on the "assessment" side Content Assessment:   Content Assessment YouTube Content Assessment Documentation Preparing for upgrade or modernization: understanding your deployment A First look at Migration from SAS  9.4 to Viya In this post, we will look at the "migration" applications which can be split into two categories, those that assist with export from SAS 9.4, and those that assist with import to Viya.   Export applications There is currently one application that helps with exporting from SAS 9.4. The createSASPackages application creates SAS packages (.spk files) for SAS objects that you intend to migrate from SAS 9 to SAS Viya.   Requirements to use createSASPackages. The SAS Content Assessment must be installed on the same machine as   SAS 9.4 Foundation SAS 9.4 Platform Object Framework The application currently supports Data Integration Studio jobs. Using the "SAS 9 Profile for SAS Data Integration Report" created by the ProfileContent application, you can create a CSV file of jobs that can then be input to the createSASPackages application.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     The createSASPackages application creates SAS packages for each object in the CSV. The packages are output in the location set in the parameter ASSESSMENT_PACKAGES_OUTPUT_LOCATION set in the setenv.yaml file.   createSASPackages.exe --source-file "D:\\workshop\\94packages\\Jobs ready for migration or not.csv"   Each job that is referenced in the CSV file, and any dependent objects, are exported to a .spk file.     Import applications There are two import applications importEGProjects and importSASPackages.   importEGProjects imports SAS Enterprise Guide projects on the file system to SAS Viya 4. importSASPackages imports SAS packages containing SAS Enterprise Guide projects and SAS Data Integration Studio jobs to SAS Viya 4.   These applications are used in conjunction with the SAS Viya Command-Line interface(CLI). They are a little different than the rest of the Content Assessment applications in that they do not access the SAS 9.4 environment. Instead, they read files (.spk and .egp) from the file system and, using the transfer plug-in of the SAS Viya CLI to import the content of those files to SAS Viya.   Requirements to use the import applications:   installed on the same machine as the SAS Viya CLI (minimum release 1.7.11) the SAS Viya CLI must include at a minimum the 1.20.0 release of the transfer plug-in. can access the Viya environment can access the directory with the SAS Package files and/or Enterprise Guide projects.   These applications are really stand-alone applications delivered with the Content Assessment Utility they are installed in the migration sub-directory of the Content Assessment install directory. They can be used independently of any Content Assessment analysis (NOTE: To use the CSV import you need a CSV file in the format created by the Content Assessment reports).   Using the importEGProjects application there are two ways to specify which projects to import. You can use a CSV file downloaded from the Content Assessment Reports or you can reference the EG projects directly.   Use --source-location to specify the path to a directory containing SAS Enterprise Guide projects. You can optionally specify a base directory to store the EG projects (the default directory is "SAS Content> ConvertedEGProjects").   ./importEGProjects.exe --sas-viya-path "D:\\ViyaHome\\CLI\\sas-viya.exe" --source-location "D:\workshop\gelcorp\finance\projects"   Check out this article to see more details on using importEGProjects with the output of the Content Assessment reports. ImportPackages works in exactly the same way except the --source-location should point to a directory that includes SAS packages.   Summary Content Assessment now delivers additional applications that can assist with the execution phase of a migration.   createSASPackages application creates SAS packages (.spk files) for SAS objects that you intend to migrate from SAS 9 to SAS Viy importEGProjects imports SAS Enterprise Guide projects on the file system to SAS Viya 4. importSASPackages imports SAS packages containing SAS Enterprise Guide projects and SAS Data Integration Studio jobs to SAS Viya 4.   I encourage you to take a look at Content Assessment on a regular basis as new features are delivered every month (this post was originally based on 2021.1.5). You can see what has changed in the Version History section of the documentation.      Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya 2021.1.4 includes a new generic approach to making a backup package available to a target environment in a SAS Viya migration or disaster recovery. The 20201.1.4 documentation includes instructions on how to use this new approach which should work with only minor differences regardless of the cloud provider of the target SAS Viya environment. In this post, I will look at this new approach   I discussed backup and restore in SAS Viya in a previous article.  In that post, I described how a Viya backup is stored on two main persistent volumes claims(PVC) within the k8s cluster (depending on what software you have deployed there could be additional volumes.):   sas-cas-backup-data contains the backup package for the CAS server including the default caslibs and the CAS permstore sas-common-backup-data contains the backup of the Viya Configuration Server and Infrastructure Data Server   You can use kubectl to see the volumes in your deployment.   kubectl get pvc -l "sas.com/backup-role=storage" -n $current-namespace   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Each individual backup within these PVC's is in a directory which is named with a DateTime stamp representing the backup id of the backup. It is these directories that represent the backup package and that must be copied to the target deployment for a restore.   How does this new approach work? To copy the backup package to the target Kubernetes cluster the kubectl cp command is used. Kubectl cp is a command that can copy files and directories to and from containers (similar to SCP in Linux).  The basic syntax (below) specifies a source and target destination for the copy operation which refer to either a local or remote destination.   kubectl cp [file-spec-src] [file-spec-dest]   The format to specify the remote destination is: <k8s-pod-name>:<file-or-dir>. As an example, to copy a single file (SAS dataset) into the CAS public directory mounted into the CAS controller you can run the following kubectl cp command.     kubectl -n {TARGETNS} cp /mnt/workshop_files/dataforpublic/hmeq_test.sas7bdat sas-cas-server-default-controller:/cas/data/caslibs/public/hmeq_test.sas7bdat   To perform the copy the file or directory must be accessible to the machine where kubectl is running (either directly on the machine or via some kind of share). You also have to be aware of permissions, both inside the pod and of the user identity you use to run the kubectl cp. Make sure:   that the directory and file permissions are the same in the target environment as in the source environment. the user that issues the  kubectl cp command to copy files has appropriate permissions to copy data from the backup package and to the target PVC.   To copy and restore a backup you need to know the backup id. The backup id is stored in a status.json file inside the backup. To identify it follow the instructions in the documentation.         Copy to the CAS backup to sas-cas-backup-data To copy the CAS backup we can follow this same approach that was used to copy the SAS dataset to the public caslib. The CAS controller pod is always running in a Viya deployment and the containers in the pod have a PVC sas-cas-backup-data  This is the location where CAS backups are stored, and also the location where we can copy CAS backups if we wish to restore from them.   To copy the CAS backup to the target deployment it must be accessible to the machine where you are running kubectl.   Step 1) Get the name of the CAS controller pod.   There are a couple of ways to do this. You can check where the sas-cas-backup-data PVC is mounted.   CONTROLLERPOD=$( kubectl -n ${TARGETNS} describe pvc sas-cas-backup-data | grep >"Mounted By" | awk '{print $3}' ) echo ${CONTROLLERPOD} sas-cas-server-default-controller   OR you can get the name of the controller using this command. The environment variable contains the name of the CAS controller pod in your deployment.   CONTROLLERPOD= $(kubectl get pod -l casoperator.sas.com/node-type=controller --output=jsonpath={.items..metadata.name}) echo ${CONTROLLERPOD} sas-cas-server-default-controller   Step 2) Use kubectl cp to copy the CAS backup to the target environment.   Use kubectl cp to copy the backup we want to the target. Use the   backup id of the backup sas-backup-agent container backup mount point in the container which is /sasviyabackup.   kubectl -n ${TARGETNS} cp /tmp/backup/cas/${backupid} ${CONTROLLERPOD}:/sasviyabackup -c sas-backup-agent   Step 3) Test that the backup package was copied. Test that the backup data has been copied. To do this exec into the pod and run an ls command to display the contents of the /sasviyabackup directory.   kubectl -n ${TARGETNS} exec -it ${CONTROLLERPOD} -c sas-backup-agent -- bash -c "ls -al /sasviyabackup" total 0 drwxrwxrwx 3 root root 42 Aug 24 20:58 . rwxr-xr-x 1 root root 76 Aug 24 20:07 .. drwxr-xr-x 3 1002 sas 25 Aug 24 20:58 2021-08-24T19_15_27_627_0700   Copy the Common Backup to sas-common-backup-data The common-backup-data is more complicated and this is where the new functionality comes in. By default, the only time that the sas-common-backup-data PVC is mounted is when a backup or a restore Kubernetes job is running. As a result, there is generally no running POD that can be used as a destination to copy the common backup data.The new functionality provided starts a Kubernetes job that contains a POD that uses sas-common-backup-data PVC. Running the job provides a POD with a target destination for a copy the common backup data.   In 20201.1.4 the overlay defining the job is in the project directory at /sas-bases/examples/backup/sas-backup-copy-job.yaml. Instructions for using it are provided in the readme.md file in the same directory and here in the documentation.   To copy your common backup data to the target environment follow these steps. Step 1) Keep a copy of the current kustomization.yaml to restore after the copy is done and, if it doesn't already exist, create a folder under site-config called backup.   cp kustomization.yaml kustomization.yaml.save mkdir -p ~/project/deploy/${TARGETNS}/site-config/backup   Step 2) Copy the sas-backup-copy-job.yaml file from /sas-bases/examples to the new ../site-config/backup sub-directory..   cp ~/project/deploy/${TARGETNS}/sas-bases/examples/backup/sas-backup-copy-job.yaml ~/project/deploy/${TARGETNS}/site-config/backup   Step 3). In the backup directory create a kustomization.yaml that includes a reference to the sas-backup-copy-job.yaml overlay. The file should look like this.     Step 4). Update the kustomization.yaml file in the project directory to reference the kustomizations in the backup directory by adding the line -site-config/backup to the resources section.       Step 5) Build the manifest that will start the job.   cd ~/project/deploy/${TARGETNS} kustomize build -o site-sas-backup-copy-job.yaml   Step 6) Start the job by applying the manifest. The job will start and a POD inside the job will mount the sas-common-backup-data PVC.   kubectl -n ${TARGETNS} apply -f site-sas-backup-copy-job.yaml -l sas.com/job-type=sas-backup-copy-job job.batch/sas-backup-copy-job created   Step 7) Get the name of the POD that the job creates so that we can use it in the kubectl cp command.   PODNAME=$( kubectl -n ${TARGETNS} get pods -l sas.com/job-type=sas-backup-copy-job -o jsonpath="{.items[0].metadata.name}" ) echo ${PODNAME} sas-backup-copy-job-8wj9p    Step 8 ) Use kubectl cp to copy the data to the target common data PVC.   kubectl -n ${TARGETNS} cp /tmp/backup/common/${backupid} ${PODNAME}:/sasviyabackup   Step 9). Test that the backup data has been copied.   kubectl -n ${TARGETNS} exec -it ${PODNAME} -- bash -c "ls -al /sasviyabackup"   You should see a directory with the name of the backup id that was copied in.   total 0 drwxrwxrwx 3 root root 42 Aug 24 20:58 . drwxr-xr-x 1 root root 76 Aug 24 20:07 .. drwxr-xr-x 3 1002 sas 25 Aug 24 20:58 2021-08-24T19_15_27_627_0700   Step 10) Lastly restore the original kustomization.yaml and delete the backup copy job.   rm site-sas-backup-copy-job.yaml mv kustomization.yaml.save kustomization.yaml kubectl -n ${TARGETNS} delete job -l sas.com/job-type=sas-backup-copy-job   With the package now available in the target PVC you can now perform a restore. This new functionality will help with copying backup packages between Viya environments. For more information check out these resources.   A first look and Backup and Restore in Viya How to use Backup and Restore of a SAS Viya enviro... - SAS Support Communities Part 1 How to use Backup and Restore of a SAS Viya enviro... - SAS Support Communities Part 2 Administration Guide: Backup and Restore Documentation: Copy Data from One Kubernetes Environment to Another   Find more articles from SAS Global Enablement and Learning here. ... View more

User-defined format in CAS and Viya was a gold mine for early GEL blogs.  This was because the way CAS accesses formats is very different than BASE SAS.  To summarize, in Viya, if a CAS table has columns that reference user-defined formats, the formats must be made available to the CAS server. CAS cannot directly read SAS catalogs that store user-defined formats. SAS format catalogs must be converted to a CAS format library and stored in a sashdat file in a path-based CASLIB. In addition, to be made available to all users, the CAS format library must be promoted in a global CASLIB. In this article, I will look at how life has become easier for the Viya Administrator in relation to user-defined formats and CAS.   In a previous article, I described some of the interfaces that an administrator can use to make user-defined formats available. The interfaces include:   SAS Environment Manager The CAS plugin of the admin/viya command-line interface SAS Code using statements, proc CAS, cCAS actions and/or proc casutil.   In previous releases, the methods above would make user-defined formats available to the CAS server, however, access to the formats would not persist across CAS server restarts. The only way to persist the availability of user-defined formats across server restarts was to add lua code to the CAS server start-up script files (casstartup.lua and casstartup_usermods.lua) and in the code:   Load the format library into memory in the global FORMATS caslib Add the format library to the CAS server format search path.   What has changed? In Viya 4  you no longer need to update the CAS startup files. Now, regardless of the interface that you use, when you perform the two steps above, access to format libraries is persisted across CAS server restarts. (Side note: changing server startup files is very different in Viya 4 as described in this article).   Let's see how it works. The CAS permstore is a directory that contains the metadata about CAS including Caslib definitions, permissions, etc. In Viya 4, the permstore is located on a persistent volume and is accessible inside the CAS controller pod at the location /cas/permstore. For more details on storage and CAS check out this article.  Persistence of access to formats now works automatically with the help of the CAS permstore. Now when you add a format library in Environment Manager or with any of the other interfaces two files are written to the permstore. These files are persisted in the permstore and used every time CAS starts to configure access to user-defined formats. To see this in action, first, let's use a kubectl command to get the name of our CAS controller pod.   kubectl get pod -l casoperator.sas.com/node-type=controller --output=jsonpath='{.items..metadata.name} {"\n"}'   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Next, we will exec into the CAS controller pod and check the contents of the permstore.   kubectl exec -it sas-cas-server-default-controller -c cas -- ls -li /cas/permstore/primaryctrl     All good so far, now let's use the CAS plug-in to the sas-viya command-line interface to: create format library import formats from a SAS catalog update the format search order   The command below performs all these tasks, AND now persists the changes in the CAS permstore. If you would like to see how to do this in Environment Manager check out my previous article.   sas-viya cas format-libraries create --server cas-shared-default --format-library slformats --search-order append --source-path /gelcontent/gelcorp/sales/formats/formats.sas7bcat --su   Let's repeat the command we used to view the contents of the permstore. In the permstore you can see two new files setfmtsearch_startup.lua and addfmtlibs_startup.lua.   kubectl exec -it sas-cas-server-default-controller -c cas -- ls -li /cas/permstore/primaryctrl     If we view the contents of the files we will see that they contain the code to make the format library available and add it to the search path. This is the same code that, in previous releases, you would have had to add to the CAS startup files as an additional step. addfmtlibs_startup.lua   kubectl exec -it sas-cas-server-default-controller -c cas -- cat /cas/permstore/primaryctrl/addfmtlibs_startup.lua     setfmtsearch_startup.lua   kubectl exec -it sas-cas-server-default-controller -c cas -- cat /cas/permstore/primaryctrl/setfmtsearch_startup.lua     These two files are now generated/updated automatically when a format library is created and/or added to the search path. They are persisted in the permstore and used to restore access to the formats when the CAS server restarts. No need any longer to update the CAS startup files. This is a really nice improvement in Viya ?????   Summary The documentation sums the change up perfectly:   "In SAS Viya, global-scope user-defined format libraries that are created, saved, and promoted by a SAS administrator, are automatically preserved across CAS server restarts, transfers, and migrations. SAS Viya 3.5 and earlier releases required the use of start-up script files (casstartup.lua and casstartup_usermods.lua) to add and promote global-scope user-defined format libraries across CAS server restarts."         Find more articles from SAS Global Enablement and Learning here. ... View more

From SAS Viya's initial release the ability to import and export content has been restricted to members of the SAS Administrators group. The possibility of opening this functionality to other groups of users has been a much-requested feature. Now with the release of Stable 2021.1.1, this feature is available. In this blog post, I will explain how it can be used.   Why? Membership of the SAS Administrators group in Viya bestows much power. SAS Administrators have access to all: functionality that is controllable through authorization rules. folders and all objects that the folders contain A common request is to have a lower level of administrator for Content Management (import/export). A set of users who do not have the full power of the Viya Administrators but are allowed to manage content using SAS Environment manager, and perform import and export tasks. Starting with stable 2021.1.1 content migration functionality is available to all authenticated users, but can also be locked down to specific users or groups of users.   Access to functionality in Viya is governed by rules. For more information on how this works, you can see this blog post. The specific Viya rules that control access to Content Migration (import/export) set the CREATE permission for authenticated users on the endpoints /transfer/importJobs and /transfer/exportJobs.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The result of the application of these rules is that all users who can log on to the Viya Environment have access to import and export. While this is good news, in reality, you may want to restrict access to a subset of users.   In our Administration Viya Environment, we have three business units(BU) in the fictional "gelcorp" organization (HR, Sales, and Finance). The users in each BU are restricted from accessing the content in the other BU's content using authorization settings on the SAS Folders.   Create Migration Administrators In this post, we will create a Migration Administrators group and restrict access to migration functionality to members of this group. We will have a member of the Migration Administrators group from each business unit so that one person from each BU can manage the content.   Update Rules We now have a new group Migration Administrators(migrationadministrators), the next step is to update the rules that determine who can access this functionality. To achieve this we update the rules for /transfer/exportJobs and /transfer/ImportJobs to change the Principal from authenticatedUsers to Migration Administrators(migrationadministrators).   This will restrict access to import/export functionality to SAS Administrators and members of the group. You can do this in SAS Environment Manager, in the Rules view locate the /transfer/importJobs Create rule edit the rule and replace the Principal authenticatedUsers with migrationadministrators. Your updated rule should look like this:     Repeat the process for the /transfer/ExportJobs create rule. The impact of the changes is that now only members of the Migration Administrators group can access import/export functionality in SAS Environment Manager or using the sas-viya transfer CLI. Test A simple test will be to log on to SAS Environment Manager as a user who is not a member of the group. The user will not be able to see the import interface or the import /export buttons. We can also test using the CLI.  For example, authenticate as a non-administrator Hugh and try to import a package. Notice you get a very clear message that the user is not authorized to perform this task.     If we authenticate as Henrik we see that, as a member of the Migration Administrators group we can perform the import.     As we noted at the start Administrators in Viya have broad access to the system. When importing as an administrator existing permissions on content can be largely ignored. Now, however, non-administrators that import content will have to be aware of the permissions on Viya folders and existing content.  In general, the importing user will need the same permissions as if they were going to create the content using the Viya User interface.   To summarize the user performing Content Migration must have the correct permissions to existing folders and content.​ Exporting must have Read access to the objects that are exported​ Importing Write access so that objects can be created, updated, or replaced:​ Add permission for the target folder​ Update permission on content that is replaced​ Caslib management privileges​ if creating new global Caslibs.   Let's look at the impact of existing permissions. In this case, we have a package that contains content within the HR area of the folder structure. Only HR users have access to these folders.     Let's see what happens if we try to import it as Fiona a Finance Migration Administrator.     As we would expect with the permission enforced the import process failed because Fiona does not have the correct permissions to create content in the /gelcontent/GELCorp/HR folder path. It is great to see this much-requested feature available to Viya 4 Administrators.  Some good news for Viya 3.5 users is that there is a plan to back-port this feature to Viya 3.5 later this year.   Find more articles from SAS Global Enablement and Learning here. ... View more

In SAS 9.4 Enterprise Guide (EG) is a very popular tool for a variety of SAS-related tasks. EG is used for data preparation, data analysis, analytics, reporting, and just about any other task you could perform with SAS code. In this article, I will look at how you can migrate your SAS 9.4 EG projects to SAS Viya 4. EG projects are migrated to SAS Studio flows.   What is Supported? The SAS Viya 2020.1.4 stable release is the first release that supporting import of EG projects. At a high level here is what is supported: EG Projects to SAS Studio Flows Projects created in Enterprise Guide 7 and later. An EG project can contain one or many Process Flows. Each Process Flow is made up of a series of tasks. When an EG project contains multiple Process Flows a Viya folder is created on import, and each flow is migrated to a SAS Studio Flow in the folder. Currently, each EG task is migrated to SAS Studio Flow Code Step.   Content Assessment SAS 9 Content Assessment can help with the migration of EG projects. SAS 9 Profile for SAS Enterprise Guide helps you identify SAS Enterprise Guide projects in your SAS 9 system. The report contains file system organization and time-based usage patterns of  Enterprise Guide projects.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SAS 9 Code Check for EG Projects helps you understand what code in your EG Projects is a candidate to migrate to SAS Viya. SAS 9 Code Check checks .sas files as well as checks results from the SAS 9 Gather SAS Code application – an application that collects SAS code stored SAS 9 metadata or embedded in Enterprise Guide projects. SAS 9 Code Check for EG Projects enables you to scan your SAS 9 system, review results, and plan for migration by understanding what code is expected to work in your SAS Viya environment.       In the report the "Compatible SAS Enterprise Guide Projects" table can be used to export a comma-separated-value file (csv) file that can drive the batch import of the projects identified as compatible. More on that later.   Import EG projects could reside on the filesystem (most common) or in metadata. You can import EG projects from both locations. There are a few options available to import your projects. To import: EG projects stored in metadata first export to a 9.4 package (.spk) file as we do with other SAS 9.4 content and then import to Viya. Individual Projects(.egp file) use the import interface in SAS Environment Manager or the transfer plugin of the sas-viya CLI to import the .egp file Multiple projects (.egp files) use the importEGProjects application delivered with SAS 9 Content Assessment. Using the importEGProjects application there are two ways to specify which projects to import.   Use a folder path Use --source-location to specify the path to a directory containing SAS Enterprise Guide projects.   ./importEGProjects.exe --sas-viya-path "D:\\ViyaHome\\CLI\\sas-viya.exe" --source-location "D:\workshop\gelcorp\finance\projects"   Use a csv file Use  --source-file  to specify the fully qualified path to a CSV file containing a list of SAS Enterprise Guide projects to import. The CSV file contains a list of compatible projects, it was created in the planning phase from the EG Code Check report.     ./importEGProjects.exe --sas-viya-path "D:\\ViyaHome\\CLI\\sas-viya.exe" --source-file "D:\workshop\94packages\EGPImportList.csv"   When importing projects in batch you can choose a root folder for the projects (the default is (SAS Content > ConvertedEGProjects). Under that folder, the full path to the project as it was on the source file system is created in the Viya folder structure. If a project contains multiple Process Flows a folder is created for that project containing a SAS Studio Flow for each process flow.     SAS Studio Flows In the Content Migration area things are changing quickly. In this first release of EG migration, each "task" in an EG process flow is converted to a SAS Program Step in a SAS Studio flow. The SAS program Step contains SAS code that performs the function of the EG task.     Future development will add support for the migration of tasks to steps with equivalent functionality e.g., query builder task to query step. Check out the Viya Administration guide for more details about specific Enterprise Guide tasks.  Some Enterprise Guide tasks: are not supported because there is no equivalent in Viya e.g., Update Library Metadata. require manual updates of the resulting program step e.g., Create Format from Dataset. If a task in a flow is not supported, it will be noted in the log and in the Environment Manager UI.   Additional Steps Resources that the EG projects need such as code, data files, and SAS datasets must be accessible to the target Viya environment. Data for EG projects migrated to SAS Studio Flows must be made available to the Viya compute server. To make data available add the libname statement to the startup of SAS Studio compute context. For details on how you can do this see this article on "Modifying Server Startup Configuration in SAS Viya". The Content Assessment reports will have given you details of where the projects are accessing resources on the file system that may be in a different location in SAS Viya.   Validation Another new feature the Enterprise Guide Migration Summary report will help you look at the results of migrating SAS 9 Enterprise Guide Projects to SAS Studio Flows in SAS Viya. Results are available for each node in EG Projects, indicating whether the resulting steps in SAS Studio Flows need attention in any way. The report is delivered with the Content Assessment Applications and the data is retrieved by the transfer cli. Use the transfer plug-in of the SAS Viya cli to generate an Import History CSV File   sas-viya transfer import-history --all -f /tmp/importHist.csv   Import the CSV file to the SystemData Caslib   sas-viya cas tables import csv --server cas-shared-default --caslib SystemData --contains-header-row --scan-rows 99999 --replace --force --table ImportedItems --source-file /tmp/importHist.csv --su   The report is located  at SAS Content > Products > SAS Environment Manager > Dashboard Items > Enterprise Guide Migration Summa. Key information includes: Status of the import of each project Migration status of steps within projects Summary by type of step and importing user     Keep in mind that this is the first release that supports Migration of EG Projects. Content Migration functionality is improving with every stable release. Expect to see additional functionality in the coming months.   For additional information: SAS Demo | SAS Viya Migration of Enterprise Guide Project to SAS Studio Flows A first look at Migration from SAS 9.4 to SAS Viya SAS Help Center: SAS 9 Content Assessment SAS Help Center: SAS Viya: Content Migration from SAS 9.4 SAS Help Center: SAS Studio Working with Flows   I would like to thank the following people for their input to this article Christiana Lycan, Gary Mehler, Amy Peters, Brian Johnson and Scott McCauley.     Find more articles from SAS Global Enablement and Learning here. ... View more

Backup and restore is always a critical feature of enterprise software. In the new SAS Viya (2020.1 and later) the architecture and implementation of this feature has changed significantly from previous releases. In this "first look" post I will provide an overview of backup and restore in the latest release of SAS Viya.   In SAS Viya 3.x backup and restore was available from SAS Environment Manager and from the backup plug-in of the sas-admin CLI. This is not the case in SAS Viya 4. In the latest release of SAS Viya backup and restore are implemented using native Kubernetes functionality. Backup and restore are implemented as Kubernetes jobs. Kuberenetes jobs are used when you want to create pods to perform a specific task to completion and then exit. Backup and restore is a common use case for jobs.   So how do we create and run these jobs? As with many administration tasks, we build and then apply a Kuberenetes manifest to make backup and restore happen. If I just lost you please check out this article for an overview of building manifests with Kuztomize.   Before we look at the process, let's look at what is backed up, what is not backed up, when it is backed up, and where it goes.   A SAS Viya backup includes:​   ​Content stored in the Infrastructure Data Server​ (reports, folders,rules, etc.) Configuration stored in the SAS Configuration Server​ (application configuration settings etc.) CAS permstore ​(Caslib definitions, access controls, etc.) Default CAS persistent volume content ​(contents of the Public caslib etc.)   Not included in the backup are the contents of your project directory (Kubernetes manifests), user home directories, other data sources such as the source of path-based caslibs, third-party databases, etc.   The default backup is a cronjob (a kubernetes job that runs on a schedule), it runs at 1 am every Sunday but you can change this schedule. Backups are written to and read from Kubernetes persistent volumes. The deployment process creates two persistent volumes:   sas-cas-backup-data sas-common-backup-data You can use kubectl to see the volumes in your deployment.   kubectl get pvc -l "sas.com/backup-role=storage" -n $current-namespace     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   It's all a bit abstract at this point so let's see how we would perform an ad-hoc backup. An ad-hoc backup can be run at any time, potentially as a backup before making a change to the system, or as the start of a migration to a new system.   Perform an ad-hoc backup To perform an ad-hoc backup job you can start the scheduled backup on an ad-hoc basis To do this use kubectl to run the following command. The command will execute the regularly scheduled backup immediately.   kubectl create job --from=cronjob/sas-scheduled-backup-job sas-scheduled-backup-job-adhoc-001 -n $current-namespace   To get the status of the backup I kubectl get jobs -l "sas.com/backup-job-type=scheduled-backup" -L "sas.com/backup-job-type,sas.com/sas-backup-job-status" -n $current-namespace   The backup package will be on the two persistent volume claims mentioned earlier   The sas-common-backup-data-pvc​  contains a sub-directory named for the unique backup id of this backup. ​ Navigating through the sub-directories you will find a status.json file that summarizes the status of the backup and two directories:   consul contains the backup of the Configuration Server​ postgres contains the backup of the Infrastructure Data Server​     The sas-cas-backup-data-pvc has a sub-directory named for the unique id of this backup (backupid).​ Within the sub-directories are two main directories.​   ​cas contains a sub-directory for each CAS server, that sub-directory contains the CAS permstore for that server. The permstore contains the casllb defintions, authorization etc.​ filesystem contains a sub-directory for each CAS server, within the sub-directory is a caslibs directory that contains the data from the SAS Viya default caslibs, e.g. public, samples, sysData, vamodels, etc.​     When the backup is complete you will want to remove the reference to the backup job from kustomization.yaml. If you don't do this it will be included in the next manifest you generate and run when that manifest is applied.   Perform a Restore Restore is similar to backup but it is a two-stage process. If you are restoring to a different system you need to have a running SAS Viya deployment to restore to, and you need to make the backup package available in the persistent volume on the target system. During the restore process the environment will be (mostly) be stopped. To restore the process requires you to create two Kuberenetes manifests that are applied in separate steps. It can be a little confusing as you create the manifests in the opposite order from which you apply them.   Create restore.yaml Restarts the services and starts the CAS server in restore mode. Edit ~/project/deploy{$current_namespace}/kustomization.yaml to define a few values in a configMapGenerator to:   SAS_BACKUP_ID to identify the backup package using the backup id SAS_LOG_LEVEL to set the log level(optional) SAS_DEPLOYMENT_START_MODE  to trigger the restore.   The following should be added to the kuztomization.yaml file.   configMapGenerator: [... previous resource items ...] - name: sas-restore-job-parameters behavior: merge literals: - SAS_BACKUP_ID=${backupid} - SAS_DEPLOYMENT_START_MODE=RESTORE - SAS_LOG_LEVEL=DEBUG   Now run the build to create the site-restore.yaml manifest   cd ~/project/deploy/${current_namespace} kustomize build -o site-restore.yaml   Create restore-initial.yaml Stops all services except SAS Configuration Server (Consul) and SAS Infrastructure Data Server (PostgreSQL) and starts the restore job​ To create restore-initial edit the kustomization.yaml adding two overlays. In the resources section add the restore overlays.   resources: - sas-bases/overlays/restore   n the transformers section add the zero replicas transform that will scale most services (except the Configuration Server and the Infrastructure Data Server to zero).   transformers: - sas-bases/overlays/restore/restore-replicas-zero-transformer.yaml   Now run the build to create the site-restore-initial.yaml manifest   cd ~/project/deploy/${current_namespace} kustomize build -o site-restore-initial.yaml   Run the Restore Process To run the restore you apply the manifests in order Step 1 apply site-restore-initial.yaml  stops most services and start the restore job which restores the Infrastructure Data Server and the Configuration server. You can check on the restore job using kubectl, the status will change to completed when it is done.   kubectl get jobs -l "sas.com/backup-job-type=restore" -L "sas.com/sas-backup-id,sas.com/backup-job-type,sas.com/sas-restore-status" -n ${current_namespace}   backup status   View the logs of the restore job.   kubectl logs -f -l "job-name=sas-restore-job" -c sas-restore-job -n ${current_namespace}   Step 2 apply site-restore.yaml. This manifest will start the: Environment services by scaling up all resources CAS Server in restore mode which will restore the data and configuration from the backup package. In order to restore the CAS file system, the restore requires a clean  CAS data volume. List the PVC for reference and then delete the persistent volume claims for CAS   kubectl get pvc -l 'sas.com/backup-role=provider' -n ${current_namespace} kubectl delete pvc -l 'sas.com/backup-role=provider' -n ${current_namespace} Check the log to see if the CAS server performed the restore. The logs should show the start of the restore process that restores the backup content to the target CAS persistent volumes.   kubectl logs sas-cas-server-cas-shared-default-controller -c cas -n ${current_namespace} | grep -A 10 "RESTORE"     Check and see if the permstore was restored. Messages should indicate that the CAS permstore is restored from the backup package.   kubectl logs sas-cas-server-cas-shared-default-controller -c cas -n ${current_namespace} | grep -B 1 -A 5 "restoring permstore"     Uses of backup and restore in SAS Viya In addition to scheduled and ad-hoc backups the backup and restore tools are also the way that SAS Viya supports migration between two SAS Viya environments and disaster recovery. I hope you have found this post a useful introduction. Here are some additional resources:   Backup and restore documentation SAS Viya to Viya Migration Documentation Backup and Recover from a Disaster Find more articles from SAS Global Enablement and Learning here. ... View more

As I noted in my previous article SAS Viya is the future of SAS, as a result making it easy to get to SAS Viya (2020.1 and later) is critical. In my last article, I looked at getting from SAS Viya 3.x to the new SAS Viya. In this article, I will review at a high-level new features available in this release to make it easier to migrate from SAS 9.4 to SAS Viya.   Content Assessment SAS 9 Content Assessment is a collection of applications designed to help you understand various characteristics of your SAS 9.4 system. The utility is intended to help customers plan for a transition to SAS Viya. Although SAS 9 Content Assessment has been around for a while now, development work is ongoing, and the included tools continue to improve. The applications that will help with content migration from SAS 9.4 are:   inventoryContent: counts the various SAS artifacts and content on your system. profileContent: profiles specific SAS artifacts and content on your system. Profile takes a deeper look at the attributes of content. codeCheck: evaluates SAS code for compatibility with SAS Viya and CAS. In addition, Content Assessment has added a few "helper" applications.   importEGProjects: imports multiple SAS Enterprise Guide projects on the file system to SAS Viya 4​ importPackages: imports multiple .spk files projects on the file system to SAS Viya 4.​   Note: a fourth application SystemEvaluation helps with Upgrading within SAS 9.4 These applications are an important part of the planning process for those embarking on the journey from SAS 9.4 to SAS Viya. I will not focus on content assessment here as there are lots of existing resources :   Content Assessment YouTube Content Assessment Documentation Preparing for upgrade or modernization: understanding your deployment This article will focus on what new 9.4 objects are supported by the content migration tools and the new interface in SAS Viya.   SAS 9.4 Supported Content With the new release of SAS, Viya support has been added for many more SAS 9.4 objects. The tables below cover the current situation. They list the SAS 9.4 content, the level of support, and what happens to the 9.4 content when it is migrated for use in SAS Viya.   Before we look at the table let’s look at how the migration of objects works in a general sense. SAS 9.4 objects that include process flows, like Enterprise Guide Projects and Data Integration Studio Flows are usually migrated to SAS Studio Flows in Viya.   It is also common for SAS 9 EBI and DI objects to be migrated into a Job in SAS Viya and/or a Visual Analytic Data View. In general, for objects that require:   Joins and filtering a SAS Viya Job Definition and possibly Job Requests are created Calculations, hierarchies, and aggregations usually SAS Visual Analytics Data Views are created   In a previous article,  I covered what the various job objects are, if you are interested in the details that article is here. Two of the main objects we see created are Job Definitions and Job Requests.   Job definitions contain metadata about a job including the code, the type of job, any parameters and/or prompts Job requests added runtime information to a job definition. Job requests make job definitions available for execution and scheduling. Visual Analytics Data Views is a reusable and shareable template for a data source. Other types of objects we see are Visual Analytics Data Views, Data Plans, SAS Studio Flows, and SAS Studio Queries. Platform Objects In terms of platform objects, the major change in the new release is the support of database libraries to database Caslib's. Support has been added for most database libraries, the full list of supported libraries is in the SAS Viya Administration documentation.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page. Visual Analytics Objects The table below covers Visual Analytics objects. A lot of this functionality has been available in SAS Viya for a while and was covered in this previous article.   In SAS Viya 3.5 only the Visual Data Builder append query was supported. In the latest release support is added for the VDB SQL and Star Schema queries. Like the Append Query, on import, the Star Schema query is converted into a SAS Data Studio Data Plan. The SQL query is converted into a SAS Viya SAS Studio query which includes code that creates a CAS Table.     Analytics, Business Intelligence, and Data Integration Objects It is in the area of Analytic, Business Intelligence, and Data Integration content where most changes have taken place. Preliminary support for stored processes was added in SAS Viya 3.5 and I covered the details in this article.   In SAS Viya, the import of two types of OLAP Cube is supported. Detailed Data Cubes are converted to a Visual Analytics Data View which is connected to a CAS table. The Data View contains the hierarchies, levels, measures, calculated items, etc.  The data must be migrated and made available to CAS separately.   Star Schema Cubes are converted to a Job definition which contains the generated FedSQL code to create an output table and a Data View associated with the table. The Data View contains the hierarchies, levels, measures, calculated items, etc. A simple conversion of relation Information Maps is supported resulting in:   Job definition, containing the code to create the target table using PROC SQL code and load it to CAS Job request, which wraps around the job definition so that it can be submitted and scheduled. Recent development work has added support for Data Integration Studio Jobs and Enterprise Guide Projects. They are both migrated to SAS Studio Flows.  For more details on Enterprise Guide Projects check out this post     New Import User Interface The latest release of SAS Viya includes an updated interface for importing content with some great new features. The import interface is available as a choice in Environment Manager, it has two tabs, one to import content and one to manage mapping sets. We can't go through all of the new features here but some highlights are:   A mapping set management interface (more on this below) The capability to selectively import content from a package The capability to relocate content during an import     New functionality allows mappings to be managed within the import area in SAS Environment Manager. Mappings are now stored within the Infrastructure Data Server as mapping sets, removing the necessity to manually edit json or yaml mapping files within a text editor. Mapping Sets can be managed separately from the import process. They can be: Loaded and Saved from/to the Infrastructure Data Server Uploaded from a JSON File Edited Merged Deleted   Summary Migration from 9.4 to SAS Viya is an area that is very important to SAS in the coming year, Development work is ongoing and support for the migration of additional 9.4 content is being added on a monthly basis.   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS servers have always used configuration files, read on server startup and used to make resources like data or files available to the server, and to set options to change behavior. In SAS Viya 2020, server startup configuration files for CAS and Compute sessions are no longer stored and maintained in operating system files. Instead they are persisted in the SAS Configuration server (Consul) and read from that location when the server is launched. In this post I will look at what this changes means for a Viya administrator.   In SAS Viya 3.x (and even SAS 9.4) users could edit "usermods" files on the file system to control server behavior and make resources available. To use the compute server as an example, two of they key "usermods" files available for customization by an administrator were:   autoexec_usermods.sas: contains SAS statements that are executed immediately after SAS starts sasv9_usermods.cfg: specifies start-up options for the server. The CAS server also had a set of lua files that were used to perform similar processing when CAS starts. For example, castartup_usermods.lua allows the administrator to make user-defined formats and global-scope caslibs available on server startup and also potentially load global-scope tables.   In the latest release of SAS Viya on Kubernetes these files are no longer available in a file system location.  For an administrator this means that instead of using a text editor to change autoexec and config options, the updates are made in Environment Manager or using the configurations plugin to the sas-viya command-line interface.   Updating Startup Files in Environment Manager   Compute Compute server startup configurations is now persisted in the Configuration Server (Consul) and the configuration is loaded to files in the /config directory inside the Compute pod when the Compute server is initialized.   In SAS Environment Manager you can edit the startup settings in the Configurations area. There are a few configuration definitions where you can make modifications that impact SAS sessions.   sas.compute.server  applied to the SAS Compute server. sas.batch.server applied to the SAS Batch server sas.connect.server applied to the SAS/CONNECT Server.   Lets look at updating the Compute Server autoexec. The changes made here would apply to all SAS sessions launched from SAS Studio.  Keep in mind that the following process also applies to the batch server and the connect spawner.   As an administrator in SAS Environment Manager navigate to the Configuration page and change the view to Definitions . Select the sas.compute.server configuration definition. There are three configuration instances that can be updated for compute.   Compute service: autoexec_code (loaded to /config/autoexec.sas) would have been autoexec_usermods.sas and contains SAS code that is run on server startup Compute service: configuration_options (loaded to /config/sasv9.cfg) would have been sasv9_usermods.cfg and contains SAS configuration options defined for the SAS Session Compute service: startup_commands: (loaded to /config/startupscript.sh) would have been sasenv_local and contains environment variables and other commands to be set or executed prior to startup of the SAS Compute server.     Edit the Compute service: autoexec_code configuration instance. In the contents property field, add the SAS code. Currently there is no syntax checking on anything entered in Environment Manager. Be careful editing the files as errors can break the server.     For compute servers there is no need to restart any services after this change. The changes will be effective when the next user launches a compute server by, for example, logging into SAS Studio. (The connect spawner does require a restart).   In addition to making server level changes for autoexec and config, changes can be made on the different Compute Contexts. To make changes at this level, In Environment Manager select Contexts > Compute Contexts, choose a Context and select Edit.   NOTE: With version 2020.1.1, configuration instances that allow start-up script modification(startup_commands) are disabled by default, but can be enabled by a Kubernetes Administrator.   CAS CAS Startup files are surfaced in the same way. CAS server configurations are persisted in Consul and the configuration is loaded to /cas/config (located in the cas container on each CAS pod) when the CAS server is initialized. For each cas server instance (in this case cas-shared-default) the following configuration settings are available.   cas-shared-default: config ( loaded to/cas/config/casconfig_usermods.lua)​ sets environment variables and/or configuration options in the CAS server and sessions.​ cas-shared-default : settings (loaded to /cas/config/cas_usermods.settings)​ sets environment variables in the CAS server and sessions.​ cas-shared-default : startup (loaded to /cas/config/casstartup_usermods.lua)​start-up scripts (load data and formats etc.) executed before a CAS users’ session connects to the CAS server (session zero).​ cas-shared-default : logconfig (loaded to /cas/config/logconfig.xml)​ sets the level of logging levels for CAS.​ cas-shared-default : delete​ is used delete/reset all configuration instances  for the CAS deployment when the CAS server is shutdown or restarted.​   NOTE: For releases 2020.1.1 and later, all the CAS configuration instances are disabled by default.To allow the SAS administrator to modify the scripts in the configuration instances, set SAS_ALLOW_ADMIN_SCRIPTS to TRUE in the sas-shared-config configMap. The CAS server does require a restart after making configuration changes. To restart the CAS server.   kubectl delete pod --selector 'app.kubernetes.io/managed-by=sas-cas-operator' -n $current_namespace   Automation What if we want to automate the process of reading and updating the startup files This is possible using the configuration plugin of the sas-viya cli as described in this post. Step 1 Read the existing autoexec file from the configuration definition and write it to a file. This step grabs the id of the configuration instance and uses it to download the configuration using jq to extract just the code from the "contents" field.   /opt/bin/sas-viya configuration configurations download -d sas.compute.server -s compute | jq -r '.items[] | select(.name=="autoexec_code") | .contents' > /tmp/compute_current_autoexec.sas   Step 2 Next create any new autoexec code in a file.   tee /tmp/autoexec_usermods.sas /dev/null << EOF libname findata '/gelcontent/gelcorp/finance/data/'; libname hrlib '/gelcontent/gelcorp/hr/data/'; EOF   Step 3 Append the two files and store the contents of the new file in an environment variable. Make sure each line has a linefeed at the end as does the environment variable.   cat /tmp/compute_current_autoexec.sas /tmp/autoexec_usermods.sas > /tmp/compute_autoexec.sas sed -i 's/$/\\n/' /tmp/compute_autoexec.sas COMPUTE_AUTOEXEC=$(cat /tmp/compute_autoexec.sas) COMPUTE_AUTOEXEC="${COMPUTE_AUTOEXEC//$'\n'/ }"   Step 4 Create the configuration JSON to include the updated code from the environment variable and then apply the update.   tee /tmp/compute-autoexec.json /dev/null << EOF { "items": [ { "version": 1, "metadata": { "isDefault": false, "services": [ "compute" ], "mediaType": "application/vnd.sas.configuration.config.sas.compute.server+json;version=1" }, "name": "autoexec_code", "contents": "${COMPUTE_AUTOEXEC}" } ], "version": 2 } EOF /opt/bin/sas-viya configuration configurations update --file /tmp/compute-autoexec.json   As you can see things have changed for the administrator in relation to the management of server startup files. Hopefully this post has given you a good overview of the changes For more information checkout: Documentation: Programing Run-Time Environment : Server Configuration Instances Documentation: CAS Configuration Properties     ... View more

The latest release of SAS Viya (2020.1 and later) is the future of SAS, and as a result, making it easy to get to this new release is critical. In this article, I will look at how SAS Viya 3.x users can "migrate" to the new SAS Viya (2020.1 and later). For customers who want to get to the new SAS Viya (2020.1) from Viya 3.x or SAS 9.4 there will be two main approaches.    Full System Migration to a new SAS Viya environment will be the main method to get to Viya 2020.1 from Viya 3.4 or 3.5. Content Migration  using the transfer service (Environment Manager or sas-admin/viya transfer) will be the supported method to get to Viya 2020.1 and later from SAS 9.4. Content Migration will be covered in a later post. In this article, I will look at Full System Migration from Viya 3.x.   Migration to SAS Viya 2020.1 and later is supported from Viya 3.4 or Viya 3.5. The diagram below shows the overall process.     Let's look at a high-level at what happens in each phase.   Plan   In the planning phase you should understand the source environment and plan for what you want to migrate and what the characteristics of your target environment will be.   The sas-admin inventory cli plays a key part in migration to SAS Viya . The inventory cli generates the "Plan and Backup Ansible playbook". The playbook is used, in conjunction with the  3.x inventory.ini file, to scan the source deployment. In a previous article I discussed running the Inventory CLI manually, the enhancement to generate and use an Ansible playbook removes much of the initial complexity and makes it easy to both scan and package the source environment.   The playbook uses the inventory file to determine what scans to perform on each machine in the environment, it aggregates the resulting data, centralizes the data, loads it to CAS and delivers the Viya inventory report. The report helps in planning the migration. The report will help you: Locate and record all Viya content Understand what data is in CAS and plan for its migration Document resource requirements for the target environment Determine readiness   The scan process also generates a set of yaml files that allow the user to customize what data and files are included in the package when the backup is run.   Backup The Ansible playbook can also be used to create a backup/migration package. The playbook uses the backup cli to package the Viya system content and configuration. Including:   SAS Infrastructure Data Server – All instances of Postgres databases SAS Configuration Server (Consul) –  User configuration stored in Consul SAS Message Broker (RabbitMQ) – Configuration only SAS Cloud Analytic Services – Access Controls and Caslib Information (CAS permstore)   In addition to the "classic" backup the playbook reads the contentselection yaml files created by the scan, and uses the inventory cli to package content from each tier in the source environment. This process can package:   Configuration files from the file system e.g. _usermods.sas System and user-defined path-based Caslibs and their content e.g. formats, models etc. Viya generated resources such astore files and other content Optionally other user selected file-system artifacts. The contentselection yaml files can be edited to customize what data is included in the migration package. SAS recommended values are set but can be modified prior to running the Backup Play.   The migration package is a set of directories on the file system at a location set at the time the playbook is run. It includes two sub-directories, one for the user content and data and another with the classic backup files.     Initial Deployment A key pre-requisite to doing a migration to SAS Viya is that a out-of-the-box deployment must be created. Changes to the Viya configuration and content should not be made in the initial deployment as the restore process will, in most cases overwrite those changes.   Restore The restore process happens to a running Viya deployment in a namespace on a Kubernetes cluster. The restore will require elevated privileges in the Kubernetes cluster. The restore user must be able to make:   changes to the Kubernetes resources (deployments, statefulsets etc.) in the cluster. the backup package accessible to nodes in the Kubernetes cluster a location for CAS data accessible to CAS in the cluster The process involves using kubectl and kustomize to modify the kubernetes objects in the cluster to restore the content from the package to the new environment. There are three main steps in the process. The major pre-requisite for each step is that the migration package is made available to the environment via some form of storage.   Step 1 Migrate CAS Configuration The restore process will create a new CAS Custom Resource definition for SAS Viya based on the CAS settings read from the source environment, and stored in the migration package.   Step 2 Restore the SAS Data Server and SAS Configuration Server (Consul) The restore job is a Kubernetes job which restores the infrastructure data server content, and consul content from the migration package. During this step Viya target environment pods are scaled to zero with the exception of the Infrastructure Data Server and SAS Configuration Server.  In addition the restore job deletes the existing CAS custom resource definition to prepare for the CAS restore.   Step 3 Restore CAS and start the environment At startup the CAS server detects that it is in migration mode and restores the CAS permstore (CASLIB definitions and authorization settings) and any user created content that was selected to be restored.   As part of the restore:   SAS Infrastructure Data Server (PostresSQL) content is overwritten. Consul properties are partially overwritten and new values are merged in.​ SAS Configuration service properties are overwritten​ CAS perm stores are overwritten.​ Data and file restores are overwritten, and new items are added in. SAS Model Manager content is overwritten.​ Analytics projects might be overwritten.   Validation Validation will be an important step when the migration is completed. An inventory scan can be run in the target environment and the scan results loaded to CAS. The results are used by the SAS Viya Comparison report to compare the source SAS Viya 3.X environment and the target SAS Viya environment. This is a good start invalidation, but as always as part of a migration, you should go through a comprehensive validation of the new environment and its content.   A Final Word This has been a very high-level introduction to migration from Viya 3.x to Viya. Find more articles from SAS Global Enablement and Learning here. ... View more

There's no place like home.   Users of SAS Studio and other SAS Viya programming clients are used to having their operating system home-directories available while they work. In previous versions of SAS Viya this is something that happened automatically with no additional configuration (at least before SAS Studio in Viya 3.4). In the new SAS Viya (2020.1 and later) this is not the case and additional configuration is needed. To make home-directories available in Viya on kubernetes there are a few moving parts. In this post we will go through the step by step process of making home-directories available to SAS Studio compute sessions in the new SAS Viya (2020.1 and later) .   Customers often have users home directories available on  some form of shared storage like an nfs server. The process to make these home-directories available in the new SAS Viya (2020.1 and later) is:   Annotate the compute pod template to tell it home directories are available and where the nfs server is located Update identities service configuration to set identifier.homeDirectoryPrefix, the path to the home-directories on nfs Configure SAS Studio for file system access.   Mount NFS directories into Compute Pods In the new SAS Viya (2020.1 and later) compute sessions are started by the launcher. The launcher Service looks for a Kubernetes PodTemplate that contains information that is used to construct a Kubernetes job request. The PodTemplate information is used to generate the container that is run in the pod. The container in the pod performs the SAS processing. The compute pod template is where we need to add the configuration to make external content available. The first change we will make is not strictly needed for the home-directories. But we will need it to access our data in stored on nfs which is located at /shared/gelcontent.   Add the shared storage mount to the POD Template Create a kustomize overlay in site-config/compute-server-add-nfs-mount.yaml that adds the volume (the path on the shared storage) and the volume mount (the location to mount the volume inside the container).   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In the kustomization yaml add the patch in the patches section and target the sas-compute-job-config.       Now we can access the data stored in /shared/gelcontent in the pod at the location /gelcontent.   Annotate the compute POD template In our environment the home-directories are on our nfs server at /shared/gelcontent/home.    Create a kustomize overlay in site-config/compute-server-annotate-podtempate.yaml that adds an annotation to the compute pod template. The annotation value is the fqdn of the nfs server.     NOTE: the ~1 in the overlay is used for the / character so that kustomize does not interpret the / as part of the path but rather part of the annotation. In the generated yaml the path will look like this launcher.sas.com/nfs-server: rext03-0135.race.sas.com   In the kustomization yaml add the patch in the patches section and target the sas-compute-job-config.     Now that kustomization.yaml is updated we can build and apply our site.yaml.   cd ~/project/deploy/gelcorp kustomize build -o site.yaml kubectl apply -f site.yaml   For more details on the kustomization process you can check out this article.   At this point we have the shared storage available to the compute PODs. But wait, there is more. The compute pod  currently only knows the nfs server not the path on the nfs server to be mounted. The path to the home-directory location is provided in the next step.   Update Identities Service Configuration On the identities service the attribute identifier.homeDirectoryPrefix must be set, once it is set the software will mount the home directory, for each user, from the nfs server specified in the annotation  by concatenating identifier.homeDirectoryPrefix with the username returned from the identities service, for example /shared/gelcontent/home/Henrik. Documentation on this overlay is here   To set the configuration property for identities in Environment Manager under Configuration > All Services select Identities and edit sas.identities. Set the identifier.homeDirectoryPrefix to the parent-path where home directories are stored in the nfs location.       This configuration change requires a restart of the identities service. To do that we can delete the identities POD and kubernetes will start a new one.   kubectl delete pods -l app=sas-identities   With these steps completed the next compute POD that is launched will have the home-directory available.   Configure SAS Studio The last step will be to update the SAS Studio configuration. In the new SAS Viya (2020.1 and later) SAS Studio cannot, by default, access the file system. SAS Studio is initially configured without access to the file system.  However, it is possible to configure Studio to access the file system, and to make users home directories available from an NFS mount. In the following steps Environment Manager is used to configure SAS Studio to do just that, these steps could also be completed in batch using the sas-viya configuration CLI.   In Environment Manager under Configuration > All Services select SAS Studio and edit sas.studio. In the dialog change:   "showServerFiles": true to allow Studio to access the file system "serverDisplayName" : "NFS gelcontent" to name the node in Studio "fileNavigationRoot": "USER" to start navigation at the users home directory Documentation of the Studio configuration settings is here. What about CAS In this blog we have dealt with Compute, this also works for CAS. There are two requirements to make the shared home-directory available to CAS.   set the identifier.homeDirectoryPrefix the user must be a member of the CASHostAccountRequired Custom Group   Does it Work? To test for the user Henrik we can copy the following program to their home-directory at /shared/gelcontent/home/Henrik/gel_launcher_details.sas. We will test for Compute and CAS by using some code to view the home-directory setting in SAS and CAS.   /* list the attributes of this compute session*/ data _null_; %put NOTE: I am &_CLIENTUSERNAME; %put NOTE: My home directory is &_USERHOME; %put NOTE: My Launcher POD IS &SYSHOSTNAME; run; /* is my CASUSER directory mounted from NFS */ cas mysession; proc cas; builtins.userinfo; table.caslibinfo / caslib='CASUSER' verbose=true; run; quit;cas mysession terminate; run;     Logon to SAS Studio as the user Henrik and check if we can access and run the Program that is in Henrik's home-directory. The result should be:   home directory available in Studio SAS program available in the home-directory Program runs and shows the home-directory setting CASUSER directory created on shared storage in home-directory NOTE: you can click on the gif to see it full size.       I hope you find this post useful.   For additional information: How do I kustomize my Viya 4 deployment? SAS Studio Configuration Reference Identities Configuration Reference   Annotate the Launcher.       Find more articles from SAS Global Enablement and Learning here. ... View more

The new SAS Viya brings a change for CAS in where users can read and write data. In this article, I will describe the change and its implications.  From a CAS server, all access to file system paths is through caslibs.  Path-based caslibs use as their source a directory on a file-system. As a security measure, CAS has the ability to limit the paths that are available to non-administrators (CAS super-users) when they create or edit a caslib. This ability is governed by the allowlist and the denylist.   These two lists are mutually exclusive, you can only have one implemented at a time. Here is what they do: • denylist: specifies a list of paths that are not available to general users • allowlist: specifies a list of paths that are available to general users.   In the prior version of SAS Viya (3.5), by default, the denylist (formerly the blacklist) is active and includes a small list of paths where users cannot create/edit caslibs. This leaves the system mostly wide-open in relation to where a user (who is authorized to do so) can create and edit path-based caslibs.   In the new SAS Viya , in order to provide a more secure deployment out-of-the-box, the allowlist is active instead of the denylist and the allowlist includes one path /cas/data/caslibs. This path in the new SAS Viya  is typically on a persistent volume used by the CAS server. What this means is that, by default users cannot create/edit caslibs in any other location.   There are two options if you want to create caslibs in a different location:   Viya administrators who has assumed the CAS super user role are not governed by the allowlist or denylist and can create caslibs that point to any available path. Viya administrators can make additional paths available by adding them to the allowlist. The new SAS Viya runs on Kubernetes so for CAS to access data on the file system, the file system location has to be made available to the CAS pod.  One of the ways to make data available to CAS is by mounting a directory into the CAS pods. In our Viya 4 administration class (soon to be released) we mount an NFS  directory to make it available to CAS. There is an example of how to do this in this article.   Essentially the end result of the NFS mount is that we make a path on the NFS server /shared/gelcontent that contains our data, available to a mount-point /gelcontent inside the CAS pod. The change to implement the default allowlist means that if we want to define a caslib in that new location we either need to do it as a super-user or add the path to the allowlist.   For example, in the prior version of SAS Viya (3.5) we could do this and it would create a caslib:   ./sas-admin cas caslibs create path --caslib salesdl gelcontent/gelcorp/sales/data --server cas-shared-default --description "gelcontent salesdl   Running the same code in the new SAS Viya (2020.1 and beyond) fails. To get an idea of what happened we need to add the --verbose option to the command   ./sas-admin --verbose cas caslibs create path --caslib alesdl /gelcontent/gelcorp/sales/data --server cas-shared-default --description "gelcontent salesdl   {"errorCode":12299,"message":"The action was not successful. (2-0-2640595)", "details":["traceId: bc84b4db0ccb1c09","path: /casManagement/servers/cas-shared-default/caslibs "ERROR: Cannot add caslib salesdl. It uses a path that is not in the allowlist .","ERROR: The action stopped due to errors.","0x887fc253:PERM_ALLOWLISTED"], "links":[],"version":2," httpStatusCode":409} The action was not successful. (2-0-2640595)   The error message is very helpful! It is an easy fix, if the account your are using is a Viya administrator you can assume the super-user role and that will allow you to create caslibs in any location that is accessible. In code above that uses the command-line interface we can just add the -su option and the caslib creation will be successful. As a superuser we can create caslibs anywhere that CAS can access.   If you do not want to always define caslibs as a super user the other approach is to add paths to the allowlist. This can be done interactively using SAS Environment Manager or using an example overlay to update the CAS Deployment definition in site.yaml.   Add to the allowlist in Environment Manager To perform the task interactively. Logon to SAS Environment Manager as an administrator.   In the Servers area: select the CAS server whose allowlist or denylist you want to access and click Superuser icon to assume the Superuser role.   Right-click on the server and select Settings.     In the Settings window make sure that the Paths List tab is selected. To modify the active list, or to switch between lists, on the right side of the Server Settings window, click the Edit icon. If you select an allowlist you can add or remove paths to the list.     To save any changes, click Save. Add to the allowlist using kustomize and kubectl As I described in my previous article, Viya administrators will have to get used to the kubectl and kustomize to administer Viya deployments. These tools allow us to update the kubernetes manifest (usually site.yaml) which describe the Viya environment. The overall flow of the process is:   Create user-defined yaml files and store them in a directory (site-config) that is a sub-directory of the location where kustomization.yaml file resides. Edit the kustomization.yaml file to reference the newly created yaml file Use kustomize build to build the Kubernetes manifest (site.yaml) Use kubectl apply to update the environment with the changes.   To help with this process SAS delivers, with the software, many example files which can be used as a starting point for making configuration changes. Fortunately there is an example for adding to the allowlist. The example is located in the project directory at "../deploy/gelcorp/sas-bases/examples/cas/configure/cas-add-allowlist-paths.yaml.   Using the example overlay as input we create a new overlay under our site-config directory and add any paths to the list. In this case we add the /gelcontent path. The path is the mount point we configured inside the POD. The target definition indicates we want to apply this patch to all objects of the type CASDeployment.     To make the overlay available to kustomize we add the following line to the transformers section of our kustomization.yaml file   [...] transformers: [... previous transformers items ...] - site-config/cas-add-allowlist-paths.yaml [... more transformers items ...] [...]   The partial kustomization.yaml looks like this:     With these changes made use kustomize build to build the Kubernetes manifest (site.yaml) and then use kubectl apply to update the environment (only the resources that have changed will be updated)   cd ~/project/deploy/gelcorp site.yaml sitebefore.yaml kustomize build -o site.yaml kubectl apply -f site.yaml   We can look at the changes made to the manifest, by comparing before and after.     Typically with a change to CAS we need a restart to pickup the changes.   kubectl delete pod --selector='app.kubernetes.io/managed-by=sas-cas-operator'   With CAS restarted the allowlist is updated and users other than the super-user can create/edit caslibs in any path in the list. While this change is not a huge one it does result in different behavior in Viya 4 and can have an impact on existing scripts and functionality. Fortunately, it is easy to recognize and fix. For more information:   Manage Path Lists (Allowlists and Denylists) How do I kustomize my Viya 4 deployment Manage CAS Role Membership     ... View more

In recent months I have been working on the enablement materials for the new release of SAS Viya. You no doubt know by now this release will only run on Kubernetes. This will mean changes for SAS Viya Administrators. One of the tools the administrator is going to have to get used to is Kustomize. In this post I will introduce kustomize and share some experiences of  using this tool and SAS Viya on Kubernetes.   There is a lot to learn about Kubernetes and SAS Viya Deployment in this blog post I am going to skip the details and keep it simple with a few key facts.   A Viya deployment in Kubernetes is made up of set of Kubernetes resources which work together to run the software in a Kubernetes cluster. To create SAS Viya on the cluster Kubernetes accepts manifests. A manifest is just one or many yaml files. Kubernetes manifest describes the resources (e.g., Deployments, Services, Pods, etc.) you want to create, and how you want those resources to run inside a cluster. When you apply the manifest Kubernetes creates or modifies the resources in the cluster. When you perform a deployment SAS delivers a directory with deployment assets. The deployment assets include the following:     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Using the kustomization yaml file the kustomize software reads the other yaml files, SAS provided and user created, ​and generates a manifest that describes the complete deployment. Applying the manifest creates your Viya deployment.   This K8S manifest describes the system and can be modified to change/update aspects of the system. ​In working on the SAS Viya training materials it quickly became apparent that updating the manifest file is not just a deployment time activity, it is also going to be a common task for SAS Viya administrators. ​   From an administration perspective the changes you might make may include: ​Making external content available​ to SAS Viya in the cluster Providing access to permanent storage Allocating or restricting resources such as memory or cpu.​ Starting/stopping resources   How do we make changes to the Kubernetes manifest after deployment? ​There are a few tools available that could help, or we could just edit it, but the tool of choice for Viya deployments is Kustomize. What is Kustomize​?   Kustomize is a tool that can be used to create or update k8s manifests. Kustomize “traverses a Kubernetes manifest to add, remove or update configuration options”.​​ Kustomize accepts a kustomization.yaml file as input and, from its content, creates/updates a Kubernetes manifest (in this post the manifest will be named site.yaml). ​So we have yaml, creating/updating yaml.   The overall flow of what an administrator would do is illustrated in the diagram below.   To summarize. Create user-defined yaml files and store them in a directory (site-config) that is a sub-directory of the location where kustomization.yaml file resides Edit the kustomization.yaml file to reference the newly created yaml file Use kustomize build to build the Kubernetes manifest (site.yaml) Use kubectl apply to update the environment (only the resources that have changed will be updated) The diagram below, illustrates the process.       But what about the details?  There are a number of ways that this can happen, some more complex than others. Lets first look at a simple example. Simple Example: Add a Persistent Volume Claim​ Using  resources in kustomize we can  define a complete Kubernetes resource definition in a new yaml file. The process will then include this resource in the generated manifest. Step 1) Define new resource in  a file migration_package_pvc.yaml. Create a new yaml file for a persistent volume claim. In this case we include in the yaml the complete definition of a Kubernetes persistent volume claim (PVC). ​     Step 2) Update kustomization.yaml to reference new resource definition file. Update the kustomization.yaml file to reference the file that defines the resource definition of the persistent volume claim. The resources section contains references to resource definitions that will be included in the resultant Kubernetes manifest.​The other sections such as transformers and patches are more complex and we will look at them later.​     Step 3) Use the kustomization.yaml to build the manifests (site.yaml) In the directory where you kusomization.yaml is located execute the command: kuztomize build - o site.yaml The site.yaml manifest file that is built will now include the definition of the new resource. The yaml from site-config/migration_package_pvc.yaml is simply included in the site.yaml manifest file.       When the manifest is applied using "kubectl apply" the persistent volume claim is created in the cluster. If you are a Base SAS program this is similar to a %include in SAS code where you just include some code from another program in your current program. More Complex Example: Adding nfs mount volume for CAS That was easy! Lets look at a more difficult example. In this case we have an NFS share that contains the data that needs to be mounted inside the CAS pod. In this more complex example, instead of adding a complete resource, we will make changes to a resource that already exists. The resource exists in the cluster and is already defined in the manifest (site.yaml). Step 1) Create a transformer definition yaml file. In kustomize transformers modify resource configuration by adding, updating or deleting fields. The transformer yaml is not a complete k8s resource definition but instructions on how kustomize should add/update k8s manifest. In the transformer you specify the target to locate the exact resource or resources that you want to update. The target subsets the resources that the transformer will operate on. Kind: CASDeployments group: viya.sas.com Name: all names Version: v1alpha1     Note: this is a pretty basic way to subset, there is a lot of powerful ways to subset and identify the Kubernetes resources to operate on.   For example this one would apply to all all Deployments that have a sas.com/deployment-base label that includes spring or go.   target:   kind: Deployment   labelSelector: sas.com/deployment-base in (spring,go)   In addition to the target you specify the patch. The patch defines the : Operation in this case add (add|update|delete) Path traverses the yaml data structure to the exact location to add the site Values are the name value pairs to add at that location. This yaml adds two “patches”. A volume and a volume mount to all CASDeployments in the site.yaml kubernetes manifest file.   Finding the location in the yaml to apply the patch might be the trickiest part of the whole process. Using an editor like Visual Studio Code with a yaml plugin can help to identify the path within the target yaml where the addition should be made. In the screenshot below selecting the location where the existing volumeMounts are defined shows the yaml path in the tool: /spec/controllerTemplate/spec/containers/0/volumeMounts/-.       Step 2) Update kustomization.yaml to reference new file. Update the kustomization.yaml file to reference the transformer file created in step 1.     Step 3) Use the kustomization.yaml to build the manifests (site.yaml) kuztomize build - o site.yaml The mainfest (site.yaml) file will now include the volume and volumeMounts in the CASDeployment resource. The definitions are added to the target resource based on the path within the yaml.     As well as transformers you will see patches used in kustomize. The two methods can achieve the same results.   The patch method uses the patches section in the kustomization.yaml file. In the patch definition specify a “row” for each operation and include:   op: is the type of operation: add. Path: the navigation path to the object in the manifest file (same as in the last example). value: is the parameters specified in name value pairs. One of the differences in the patch method is that you define the target in the kustomization.yaml . The result in the end is the same.     We will need to know yaml and kustomize fairly well to be able to administer a SAS Viya deployment on k8s. The deployment assets delivered by R&D provide a good set of examples including readme files to get you started.   This post has just scratched the surface. I am sure there are others that have more experience in this area. If you have experiences to share please feel free to comment.   For more information check out the following resources:   The Interdependence of SAS Viya and Kubernetes Administrators Kustomize Kustomize Examples Search for more content from our group: SAS Global Enablement & Learning.   ... View more