PostgreSQL is an open-source database management system that SAS provides uses as the SAS Infrastructure Data Server. The SAS Viya platform uses this component to store user content. The SAS Viya platform can use either an internal instance of PostgreSQL that it provides or an external instance. The PostgreSQL data transfer tool allows you to switch between these two options. This post demonstrates the process of switching a Viya deployment from an internal PostgreSQL instance to an external instance using that Azure Flexible PostgreSQL server.   The process we will follow contains the following steps.   Perform Prerequisites Prepare to use the PostgreSQL Data Transfer Tool Transfer data using the PostgreSQL Data Transfer Tool Switch SAS Viya to point to External PostgreSQL Validate that that switch worked Delete internal PostgreSQL.   This process is documented in the SAS Viya Operations guide. Please check out the documentation for the specifics of each cloud provider and offering. In this post, we will demonstrate the process for the SAS Infrastructure Data Server with SAS Viya Visual Analytics on Azure.   Perform prerequisites and prepare to use the PostgreSQL Data Transfer Tool   If you deployed with the internal instance of PostgreSQL it uses Crunchy PostgreSQL. Two versions of Crunchy PostgreSQL are used by SAS Viya. The steps in the process vary based on the version used in the deployment. Run the following code to get your software release to determine what version of Crunchy PostgreSQL is being used in your SAS Viya deployment. SAS Viya 2022.10 and later use Crunchy PostgreSQL 5, prior releases use version 4. The output below shows that we are using Crunchy version 5. The rest of the process here applies to version 5. If you have an earlier version of Crunchy check out the documentation.   kubectl get cm -o yaml | grep 'SAS_CADENCE_DISPLAY_NAME'   Expected output:   SAS_CADENCE_DISPLAY_NAME: Stable 2024.01 {"apiVersion":"v1","data":{"SAS_BUILD_TYPE":"x64_oci_linux_2-docker","SAS_CADENCE_DISPLAY_NAME":"Stable 2024.01",....   It is recommended that we have a database administrator user in the target instance of the same name as the current SAS Infrastructure Data Server administrator. The default for SAS Viya is dbmsowner. To check that this is the name on the current deployment run the following code:   kubectl get secrets sas-crunchy-platform-postgres-pguser-dbmsowner -o template={{'.data.user'}} | base64 -d   Expected output:   dbmsowner   Some of the upcoming steps will be performed from inside the crunchy PostgreSQL primary POD. To get the primary POD name and exec into the POD run the following:   export SOURCE_CLUSTER=sas-crunchy-platform-postgres PRIMARY=$(kubectl get pod --selector="postgres-operator.crunchydata.com/role=master,postgres-operator.crunchydata.com/cluster=$SOURCE_CLUSTER" -o jsonpath='{.items[*].metadata.name}' | awk '{print $1}') echo Primary Crunchy POD is $PRIMARY kubectl exec -it $PRIMARY -c database -- bash   Expected output:   Primary Crunchy POD is sas-crunchy-platform-postgres-00-c992-0   The next few steps will be performed from within the Crunchy PostgreSQL primary POD. Firstly, let us check the size of the existing database. To account for the future growth of your PostgreSQL database, SAS recommends that your new external instance has twice the space that your existing internal instance uses.  To determine the size of the existing PostgreSQL database from within the crunchy primary POD run this code:   psql -c "select pg_size_pretty ( pg_database_size('SharedServices'));"   Expected output:   pg_size_pretty ---------------- 377 MB (1 row)   The external instance of PostgreSQL must meet certain requirements. In addition to an admin user matching the name of the current user, the instance must have the appropriate extensions enabled. In the following steps, we will update the configuration of External PostgreSQL. To perform these steps, we need to know the connection information for the Azure Flexible PostgreSQL server. Using that connection information, we will connect to the target instance from inside the primary Crunchy PostgreSQL POD. The connection information is stored in the following environment variables.   TARGET_HOST_NAME="sasgnn-p21077-default-flexpsql.postgres.database.azure.com" TARGET_HOST_PORT="5432" TARGET_USER_NAME="pgadmin" TARGET_USER_PASSWORD="Aa0EmhCxZJYOJWBMdEf3uj1zr60" DATASERVER_NAME="sas-platform-postgres"   Firstly, let's create the dbmsowner user in the Azure Flexible PostgreSQL server instance and allocate the user the required privileges.   PGPASSWORD=$TARGET_USER_PASSWORD psql -h $TARGET_HOST_NAME -p $TARGET_HOST_PORT -U $TARGET_USER_NAME -d postgres -c "CREATE USER dbmsowner" PGPASSWORD=$TARGET_USER_PASSWORD psql -h $TARGET_HOST_NAME -p $TARGET_HOST_PORT -U $TARGET_USER_NAME -d postgres -c "ALTER USER dbmsowner CREATEDB" PGPASSWORD=$TARGET_USER_PASSWORD psql -h $TARGET_HOST_NAME -p $TARGET_HOST_PORT -U $TARGET_USER_NAME -d postgres -c "ALTER USER dbmsowner CREATEROLE" PGPASSWORD=$TARGET_USER_PASSWORD psql -h $TARGET_HOST_NAME -p $TARGET_HOST_PORT -U $TARGET_USER_NAME -d postgres -c "ALTER USER dbmsowner WITH PASSWORD '$TARGET_USER_PASSWORD';" PGPASSWORD=$TARGET_USER_PASSWORD psql -h $TARGET_HOST_NAME -p $TARGET_HOST_PORT -U $TARGET_USER_NAME -d postgres -c "GRANT CONNECT ON DATABASE postgres TO dbmsowner;" PGPASSWORD=$TARGET_USER_PASSWORD psql -h $TARGET_HOST_NAME -p $TARGET_HOST_PORT -U $TARGET_USER_NAME -d postgres -c "GRANT azure_pg_admin TO dbmsowner;"   Now let's enable the extensions on the Azure PostgreSQL flexible server instance. In this step, we use the Azure CLI to enable these extensions and then restart the database server.   az account set --subscription ${SUBSCRIPTIONID} az postgres flexible-server parameter set --resource-group ${RG} --server-name sasgnn-p21077-default-flexpsql --name azure.extensions --value LTREE,PG_CRON,PG_STAT_STATEMENTS,PG_TRGM,PGAUDIT,PGCRYPTO,PLPGSQL az postgres flexible-server parameter set --resource-group ${RG} --server-name sasgnn-p21077-default-flexpsql --name shared_preload_libraries --value PG_CRON,PG_STAT_STATEMENTS,PGAUDIT az postgres flexible-server restart --resource-group ${RG} --name sasgnn-p21077-default-flexpsql --subscription ${SUBSCRIPTIONID}   The target instance is now ready for data transfer. It is recommended that you perform a SAS Viya Backup before continuing with the process.   Prepare and then Transfer Data using the PostgreSQL Data Transfer Tool   The PostgreSQL Data Transfer Tool is available for download from support.sas.com. The tool is delivered in a ZIP file and is installed by simply unzipping it to a directory of your choice. The tool is implemented as a Kubernetes job and the job is configured by updating the provided configuration files and manifests.   SAS provides a script to determine the values of the required variables and insert the values into the manifest files of the tool. The script requires that mirror manager is deployed and that you have access to the license files in the SAS Viya deployment assets. In this step run the script to see the required values.   NOTE: if you have CDS PostgreSQL data you need to perform these steps for that database also.   cd /home/cloud-user/project/deploy/pgtransfer/scripts/ ./manifest-values.sh --namespace gelcorp --mirrormgr /opt/mirrormgr/mirrormgr --deployment-data /home/cloud-user/project/deploy/license/SASViyaV4_${GELLOW_ORDER}_certs.zip   Expected output:   Token values SAS-CERTFRAME-IMAGE-NAME = cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.49.1-20240105.1704477691165 SAS-CUSTOMER-PROVIDED-CA-CERTIFICATES-CONFIGMAP-NAME = sas-customer-provided-ca-certificates-6ct58987ht SAS-IMAGE-PULL-SECRETS-NAME = sas-image-pull-secrets-b7bt458c8f SAS-INGRESS-CERTIFICATE-SECRET-NAME = sas-ingress-certificate SAS-POSTGRES-DATA-TRANSFER-IMAGE-NAME = cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-postgres-data-transfer:1.6.0-20240201.1706816594189 STORAGE-CAPACITY (cds) = STORAGE-CAPACITY (platform) = 256Gi   If we are happy with the required values, we can run the script using the --apply parameter to update the PostgreSQL Data Transfer Too manifests and insert the required values.   cd /home/cloud-user/project/deploy/pgtransfer/scripts/ ./manifest-values.sh --namespace gelcorp --mirrormgr /opt/mirrormgr/mirrormgr --deployment-data /home/cloud-user/project/deploy/license/SASViyaV4_${GELLOW_ORDER}_certs.zip --apply   Expected output:   INFO: Skipping substitutions for cds - it was not found in the SAS Viya platform deployment. INFO: Applying changes to '/home/cloud-user/project/deploy/pgtransfer/manifests/platform/pvc.yaml'. INFO: Applying changes to '/home/cloud-user/project/deploy/pgtransfer/manifests/platform/job.yaml'.   Before running the tool to transfer data populate the user_input.yaml. The user_input.yaml is located in the install directory of the tool at /manifests/platform/user_input.yaml. The key information in this file is the connection information to the target database server.   The tool can also be run in phases using the "run" options.   For example, if you want to only run the pre-check steps to check that the target instance meets the requirements for transfer set run.pre-check to true and the other two run variables to false. If any of the run phases are marked false, then in order to run that phase later, you must delete the job, modify the user_input.yaml file, and re-run the Job.   target: database: "postgres" admin: "dbmsowner" password: "${TARGET_USER_PASSWORD}" fqdn: sasgnn-p21077-default-flexpsql.postgres.database.azure.com port: 5432 sslmode: verify-full run: pre-check: true transfer: true post-check: true overwrite: false status_update_timespan: 15m00s sleep_after_completion: 0m00s log_level: info   When you are ready to transfer data, you must stop the SAS Viya environment.  The following code will kick off the SAS Viya stop job and wait for the PODS to terminate.   kubectl create job sas-stop-all-`date +%s` --from cronjobs/sas-stop-all RUNNING_PODS=$(kubectl get pods --field-selector=status.phase=Running -o go-template='{{len .items}}') until [ $RUNNING_PODS -lt 2 ]; do \ sleep 5; echo Running Pods: $RUNNING_PODS; \ RUNNING_PODS=$(kubectl get pods --field-selector=status.phase=Running -o go-template='{{len .items}}'); \ done   Expected output:   Running Pods: 132 Running Pods: 87 Running Pods: 65 Running Pods: 31 Running Pods: 9 Running Pods: 4   Now we can apply the manifests to run the data transfer job. The PostgreSQL Data Transfer Tool uses two standard PostgreSQL utility programs: pg_dump to back up data from your source instance and pg_restore to load the backup to your target instance. During execution, the transfer tool performs the following tasks:   Starts the internal PostgreSQL server if it is not already running. Performs a logical backup of each database in the source server. Restores the logical backup into the target external PostgreSQL cluster, creating or re-creating the databases as necessary.   To start the transfer job, apply the mainfests.   cd /home/cloud-user/project/deploy/pgtransfer kustomize build manifests/platform -o platform-transfer.yaml kubectl apply -f platform-transfer.yaml   Expected output:   serviceaccount/sas-platform-postgres-data-transfer-sa created role.rbac.authorization.k8s.io/sas-platform-postgres-data-transfer-role created rolebinding.rbac.authorization.k8s.io/sas-platform-postgres-data-transfer-rolebinding created secret/sas-platform-postgres-data-transfer-input-f7k8hm7khk created persistentvolumeclaim/sas-platform-postgres-data-transfer-backups created job.batch/sas-platform-postgres-data-transfer created   To check the success of the job, review the log. The log should contain messages indicating the successful execution of the data transfer tool. You can grep for specific text like "successfully copied database".   kubectl logs job/sas-platform-postgres-data-transfer   Expected output:   status=success duration=515ms level=0 logger=summary ts=2023-10-05T15:47:55.015215518Z msg="pre-check sasgnn-p21077-default-flexpsql.postgres.database.azure.com | check server versions" status=success duration=15.3ms level=0 logger=summary ts=2023-10-05T15:47:55.015274819Z msg="pre-check sasgnn-p21077-default-flexpsql.postgres.database.azure.com | check shared preload libraries" status=success duration=3.95ms matchingEssentialSharedPreloadLibraries=2 level=0 logger=summary ts=2023-10-05T15:47:55.015286719Z msg="pre-check sasgnn-p21077-default-flexpsql.postgres.database.azure.com | check extensions availability" status=success duration=381ms level=0 logger=summary ts=2023-10-05T15:47:55.015295719Z msg="transfer sas-crunchy-platform-postgres" status=success duration=1m3.2s level=0 logger=summary ts=2023-10-05T15:47:55.015325319Z msg="transfer sas-crunchy-platform-postgres | backup SharedServices to /backups/sas-crunchy-platform-postgres/SharedServices" status=success duration=36.7s level=0 logger=summary ts=2023-10-05T15:47:55.015372120Z msg="transfer sas-crunchy-platform-postgres | restore from /backups/sas-crunchy-platform-postgres/SharedServices" status=success duration=26s level=0 logger=summary ts=2023-10-05T15:47:55.015394720Z msg="post-check SharedServices" status=success duration=14.3s verifiedDatabaseAttributes=3 totalVerifiedRoleDetails=5789 verifiedRoles=1 totalVerifiedTableColumns=6840 totalVerifiedTableIndexes=1027 tablesWithMatchingNumbersOfRows=821 tablesWithMatchingNumbersOfConstraints=821 totalVerifiedTables=821 verifiedSchemas=84   NOTE: If the transfer fails you can delete the job using kubectl delete job/sas-platform-postgres-data-transfer and run it again using kubectl apply.   Switch SAS Viya to point to External PostgreSQL   The data has been copied to the external PostgreSQL instance, but SAS Viya is still using the internal instance. Once we are sure that the data has been transferred successfully, we can switch the SAS Viya deployment over to use the new external PostgreSQL instance. Follow the external PostgreSQL Configuration steps here to do this, do not remove the entries for your internal PostgreSQL clusters yet. They are removed later when the internal cluster is deleted. To summarize the process.   Copy /sas-bases/examples/postgres/postgres-user.env to $deploy/site-config/postgres/platform-postgres-user.env  and add the PostgreSQL database user (dbmsowner) and password. Add a secret generator that refers to the file to. the secretGenerator block of the kustomization.yaml Copy the /sas-bases/examples/postgres/dataserver-transformer.yaml/dataserver-transformer.yaml from to the $deploy/site-config/postgres/ and edit the file to update the values. Add the file to the transformers section of the kustomization.yaml.  Here is an example of the of the dataserver-transformer.yaml.   apiVersion: builtin kind: PatchTransformer metadata: name: sas-platform-postgres-dataserver-transformer patch: |- - op: replace path: /spec/users/0/credentials/input value: secretRef: name: platform-postgres-user usernameKey: username # For internal use, do not modify passwordKey: password # For internal use, do not modify - op: replace path: /spec/registrations/0/host value: sasgnn-p21077-default-flexpsql.postgres.database.azure.com - op: replace path: /spec/registrations/0/port value: 5432 - op: replace path: /spec/ssl value: true target: group: webinfdsvr.sas.com kind: DataServer version: v1beta1 name: sas-platform-postgres   Now that the manifests are updated you can build and apply using your chosen deployment method.   Now we will validate that the Viya Environment points at the Azure flexible PostgreSQL server by checking the database registration in the SAS Configuration server. The serviceAddress in the output should match the name of your target PostgreSQL database, for example: sasgnn-p21077-default-flexpsql.postgres.database.azure.com   WARNING: Do not continue to the next step until you have validated that the Consul registration has switched to external PostgreSQL.   HTTP_PROTOCOL=https CONSUL_SVC_NAME=sas-postgres kubectl exec -it sas-consul-server-0 -c sas-consul-server -- bash -c "export CONSUL_HTTP_ADDR=$HTTP_PROTOCOL://localhost:8500; /opt/sas/viya/home/bin/sas-bootstrap-config catalog service $CONSUL_SVC_NAME"   Expected output:   { "items": [ { "address": "sasgnn-p02189-default-flexpsql.postgres.database.azure.com", "createIndex": 26687, "datacenter": "viya", "modifyIndex": 26687, "node": "sas-postgres", "nodeMeta": null, "serviceAddress": "sasgnn-p21077-default-flexpsql.postgres.database.azure.com", "serviceEnableTagOverride": false, "serviceID": "sas-postgres", "serviceMeta": {}, "serviceName": "sas-postgres", "servicePort": 5432, "serviceTags": [ "postgres", "public", "data-service-postgres", "ssl" ], "serviceWeights": { "passing": 1, "warning": 1 }, "taggedAddresses": null } ] }   If the serviceAddress still points to the internal instance or there are any other issues you can review the data server operator custom resource and logs.  There should be no errors in the data server operator logs and the operator definition should point to the new azure PostgreSQL flexible server.  If you find errors check your overlays and re-build and apply.   kubectl describe dataservers.webinfdsvr.sas.com kubectl logs -l app=sas-data-server-operator   To complete the process, restart the SAS Viya Platform by running the stop and then the start job and wait for SAS Viya to start.   kubectl create job sas-stop-all-`date +%s` --from cronjobs/sas-stop-all RUNNING_PODS=$(kubectl get pods --field-selector=status.phase=Running -o go-template='{{len .items}}') until [ $RUNNING_PODS -lt 2 ]; do \ sleep 5; echo Running Pods: $RUNNING_PODS; \ RUNNING_PODS=$(kubectl get pods --field-selector=status.phase=Running -o go-template='{{len .items}}'); \ done kubectl create job sas-start-all-`date +%s` --from cronjobs/sas-start-all kubectl wait --for=condition=ready pod --selector='app.kubernetes.io/name=sas-readiness' --timeout=2700s   If you are satisfied that the data has been transferred successfully and SAS Viya is functioning, then you can stop the internal PostgreSQL instance. Shut down the internal PostgreSQL.   kubectl patch postgresclusters sas-crunchy-platform-postgres --type json --patch '[{"op":"replace", "path": "/spec/shutdown", "value": true}]'   Expected output:   postgrescluster.postgres-operator.crunchydata.com/sas-crunchy-platform-postgres patched   Validate   Before removing the internal PostgreSQL instance, it is important to validate that Viya is functioning and that the content and configuration has been transferred.  In this step we use the ValidateViya tool to execute some validation tests.    python3 /opt/pyviyatools/validateviya.py -c /mnt/workshop_files/initenv/validation/tests.json -d /tmp/testresults -o report-full   Review the HTML Report Output in /tmp/testresults. The reports should show the folders where content was loaded (gelcorp and gelcontent) and the caslibs are all available.     Delete internal PostgreSQL.   Now we can remove the internal PostgreSQL. A script is provided to delete internal PostgreSQL servers and its Persistent Volume Claims(PVCs). Answer yes when prompted.   chmod 755 ~/project/deploy/pgtransfer/scripts/sas-postgres-delete.sh ~/project/deploy/pgtransfer/scripts/sas-postgres-delete.sh --namespace gelcorp   Finally, you can remove all references to the internal PostgreSQL from the kustomization.yaml.   ansible localhost -m lineinfile -a "path=~/project/deploy/${GELLOW_NAMESPACE}/kustomization.yaml regexp='crunchy' state='absent'"   If all the Viya environments in your cluster have been switched to using an external PostgreSQL you can delete the Crunchy data cluster-wide resources.   kubectl delete crd pgupgrades.postgres-operator.crunchydata.com kubectl delete crd postgresclusters.postgres-operator.crunchydata.com   Wrap Up   The PostgreSQL Data Transfer Tool for the SAS Viya Platform allows you to move content from an internal instance of PostgreSQL to an external instance. The transfer tool only copies the data in the internal PostgreSQL instance to an external PostgreSQL instance. Required post-transfer steps change the connection information so that the existing SAS Viya platform deployment uses the external instance instead of the internal one. Later steps remove the internal instance. In this post we have walked through the process of switching from an internal PostgreSQL instance to an external instance using Azure Flexible PostgreSQL server.     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya backup and restore has recently introduced some exciting new features that offer greater flexibility in backing up and restoring SAS Viya Environments. In this post, I will provide an overview of these features, and discuss the advantages and disadvantages of each approach. This will help you determine which backup and restore strategy best suits your requirements.   As a reminder, the tools are commonly used in three main scenarios: restoring in place, SAS Viya 4 to 4 migration, and disaster recovery. Let’s start by taking a look at the classic backup and restore functionality that we have had since SAS Viya 4 was released.   SAS Viya Application Backup and Restore   Backup with SAS Viya application backup is the default. This is the functionality we have had available since the release of SAS Viya.   A backup includes a complete backup of PostgreSQL databases, Configuration Server, and CAS metadata and data. The PostgreSQL backup and restore are performed using pg_dump and pg_restore. These utilities make a complete copy of the database into and out of the backup package. CAS data and metadata are likewise copied in full to and from the package.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Backups can be used for disaster recovery, SAS Viya 4 to 4 migration, and in-place restores. While this method is versatile and can be used in all scenarios, it may run more slowly when working with large amounts of PostgreSQL data. Similarly, the CAS backup will be slower when the default caslibs contain large volumes of data.   Backup and Restore Using Provider-Specific Processes for PostgreSQL in addition to SAS Viya Application Backup   You can exclude the PostgreSQL data from the backup process and use the PostgreSQL provider to manage that part of the backup. To do this set the environment variable INCLUDE_POSTGRES=False in the backup configMap. A full backup excluding PostgreSQL includes a complete backup of the Configuration server, CAS metadata and data, but does not include any of the PostgreSQL databases.   The PostgreSQL database backup is managed by the PostgreSQL provider. No PostgreSQL data is written to the backup package. For more details on this approach check out this article which covers its usage on Azure using Azure Flexible PostgreSQL server. This approach can also be used in conjunction with the internal PostgreSQL server using the pgBackRest utility of Crunchy PostgreSQL.     The PostgreSQL provider must support Point-in-time restore (PITR). In addition, the retention period of the PostgreSQL backup and the SAS Viya backup should be synchronized to ensure both parts of the backup are available at the point in time selected for restoration. All backup and restore scenarios are supported, except in cases where internal Crunchy PostgreSQL is used; in these cases, only in-place restore is supported.   Both the backup and the restore process are significantly faster when using this approach. The backup PVC’s will require less storage space since no PostGresSQL data is included.   Backup and Restore Using Provider-Specific Processes for PostgreSQL and Incremental SAS Viya Application Backup   A recent addition to backup and restore functionality added the ability to incrementally backup CAS data. An incremental backup strategy involves storing only the data that has been modified since the last backup was taken. The first backup in the series is a complete backup, and the subsequent backups only store the modified or new data, which is the incremental differences. This approach helps to save time in performing frequent backups.   There are two parts to the CAS backup: the CAS metadata (permstore) and the CAS data in the default caslibs. This approach creates a full backup of CAS metadata but an incremental backup of CAS data.   To use this approach, you must: Enable the Viya incremental backup job. Use Provider-Specific Processes for PostgreSQL by specifying INCLUDE_POSTGRES=false The incremental backup job, when enabled, runs every day (except Sunday) at 6 am. The retention period of the PostgreSQL backup and the SAS Viya backup should be synchronized. In addition, to perform incremental backups, a periodic full SAS Viya backup is required.   Incremental backups are supported for all scenarios except when internal Crunchy PostgreSQL is used. Again, in the latter case only in-place restore is supported.  When using this approach there are significant improvements in the speed of PostgreSQL and SAS Viya backups, especially if you have a lot of data in the databases and/or the default caslibs. In addition, the volume of data stored in the backup PVC’s will be a lot smaller.   Comparison of Backup and Restore Approaches   The table below summarizes the 3 approaches.       The recent changes mean that for all scenarios your backup package looks a little different. The format of the backup ID has changed. The backup ID is a date timestamp value that uniquely identifies each backup.   The SAS-BACKUP-ID of the full Viya backup is in the format of YYYYMMDD-HHMMSSF (the F as it signifies that this is a full backup). For example, 20240228-160410F   The SAS-BACKUP-ID of an incremental backup is in the format YYYYMMDD-HHMMSSF_YYYYMMDD-HHMMSSI. For example, 20240228-160410F_20240301-120310I. The incremental backup ID's first timestamp refers to the last full backup, and the second to the last incremental backup.     In both cases, YYYY represents the year, MM represents the month, and DD represents the day of the month. HH represents the hour, MM represents the minute, and SS represents the second. All times are in UTC.   For SAS Viya 4 to 4 migration and disaster recovery, you must copy all backup packages, including one full backup and all related incremental backups, from the source to the target.   Wrap Up   SAS Viya has recently been updated to improve its backup and restore functionality, providing more flexibility in how the process is managed. With the new changes, it’s now possible to perform point-in-time restores of PostgreSQL and incremental backups of the CAS filesystem. This allows for more efficient use of resources and faster processing of backup and restore. If you’re not concerned about time and resource usage, the existing backup and restore process remains a good option that’s easy to use.   I would like to thank Dawn Schrader, Penny Downey, JK Min and Scott McCauley for their valuable input to this post.   Please check out the Backup and Restore documentation for more details:   Backup and Recovery: Workflows Perform a Restore Using Only SAS Viya Processes for All Content Perform a Restore Using Provider-Specific Processes for PostgreSQL Schedule an Incremental Backup Perform an Incremental Ad Hoc Backup Restore PostgreSQL Cluster from pgBackRest Backup Completely or Point-In-Time-Recovery (PITR)   Check these related blog posts:   Restore a Viya Backup with SAS Viya Backup and Database Point in Time Restore | Global Enablement & Learning Blog SAS Viya: Restore a Backup in 2023.10 and later. | Global Enablement & Learning Blog A first look and Backup and Restore in Viya 4 | Global Enablement & Learning Blog (sas.com)   Find more articles from SAS Global Enablement and Learning here. ... View more

Traditionally SAS Viya has relied on internal processes to perform backup and restore. A recent change has added support for using third-party functionality to backup the SAS Infrastructure Data Server portion of the Viya deployment. In this post I will introduce this new feature and demonstrate how you can use it, using a SAS Viya deployment on Azure Kubernetes Service (AKS).   It is important to note that this is an addition of functionality, not a replacement. Instead of a single workflow that uses SAS Viya application backup tools to perform a backup and restore, there are now two workflows available:   SAS Viya Application Backup Only SAS Viya Application Backup and Cloud-Provider for external PostgreSQL ​   To use the second workflow the PostgreSQL provider must support Point-In-Time-Recovery (PITR) of the external PostgreSQL server to a custom restore point based on date and time. Many cloud-provider databases do provide this functionality. For example, the Azure Doc states:   “Azure Database for PostgreSQL - Flexible Server automatically performs regular backups of your server. You can then do a point-in-time recovery (PITR) within a retention period that you specify.”   Check out this page for more details on Azure Database for PostgreSQL - Flexible Server   The benefit of using PITR is that it can be significantly faster to perform a restore than the traditional SAS Viya Backup which uses pg_dump and pg_restore The traditional approach can be very slow when there is a large volume of data in the PostgreSQL database.   Let's look at how to backup and restore using SAS Viya application backup and Azure Database for PostgreSQL - Flexible Server PITR.   Backup   To rely on the Azure Flexible PostgreSQL server point-in-time restore functionality for the SAS Infrastructure Server backup we will set a parameter that will exclude the Infrastructure Data Server from the SAS Viya Application backup. To exclude the SAS Infrastructure Data Server from the backup add the parameter INCLUDE_POSTGRES='false' to the sas-backup-job-parameters configmap. Add the following code to the configMapGenerator section of kustomization.yaml and perform a build and apply.   configMapGenerator: [... previous resource items ...] - name: sas-backup-job-parameters behavior: merge literals: - INCLUDE_POSTGRES="false" [...]   With the backup configMap updated, we can now run an ad-hoc backup. In this step, we start the backup job from the scheduled backup cronjob and give it the name sas-scheduled-backup-job-adhoc-001.   kubectl create job --from=cronjob/sas-scheduled-backup-job sas-scheduled-backup-job-adhoc-001   Follow the log of the backup job until it is completed.   kubectl logs -f job/sas-scheduled-backup-job-adhoc-001 -c sas-backup-job | gel_log   Check the job log to make sure the backup ran successfully.    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   NOTE: in the previous command we pipe to a custom function(gel_log) that uses JQ to reformat the log from JSON to a more human-readable format.   If we review the backup package in sas-common-backup-data PVC there will be no /sas-common-backup-data/${backupid}/default/postgres directory because the SAS Viya Infrastructure data server PostgreSQL database was not included in the backup. The diagram below shows the overall process and result.     Restore   Prepare for Restore   The first step in the restore is to get the backup ID of the backup to restore. Each backup is identified using a timestamp value called the backup ID. When performing a restore that includes PITR we use the timestamp value to identify :   The SAS Viya backup package AND Restore "point-in-time" within the PosgtGresSQL backup.   In this step, retrieve the backup ID from the job that created the backup.   backupid=$(yq4 eval '(.metadata.labels."sas.com/sas-backup-id")' <(kubectl get job sas-scheduled-backup-job-adhoc-001 -o yaml)) echo Backup Id is: $backupid   Output:   Backup Id is: 20240104-150710   Recently the format of the SAS Viya backup ID changed. The new format of the backup ID is YYYYMMDD-HHMMSSF (the F can be ignored as it signifies that this is a full backup) In YYYYMMDD, YYYY is an integer that represents the year, MM is an integer that represents the month of the year, and DD is an integer that represents the day of the month. In HHMMSS, HH is an integer representing the hour, MM is an integer representing the minute, and SS is an integer representing the second. All time is UTC.   NOTE: the new backup ID format is backward compatible with the old format.   The restore point format for Azure Postgres PITR is YYYY-MM-DDTHH:MM: SSZ eg. "2021-05-05T13:10:00Z". You can use this command to convert the backup ID to the format of the restore point.   restorePoint=${backupid:0:4}-${backupid:4:2}-${backupid:6:2}T${backupid:9:2}:${backupid:11:2}:${backupid:13:2}Z:${backupid:12:2}Z echo Restore Point is: $restorePoint   Output:   Restore Point is: 2024-01-04T15:07:10Z:71Z   An important feature of PITR is that the process does not restore to the same PostgreSQL server instance. To restore you need to provide a new instance name, which should be unique. It's a good practice to have a naming convention for your instances as you may end up with multiple instances in a cluster. For our purposes, we will just prepend the backup ID to the existing Flexible PostgreSQL server name eg. ${backupid}-blitz-0331-rg-default-flexpsql. In this step, we create an environment variable with the new name.   RESTORE_DB_NAME=d${backupid,,}-${GELENABLE_PREFIX}-default-flexpsql echo Restored DB instance: ${RESTORE_DB_NAME}   Output:   Restored DB instance: d20240104-150710f-boxer-p02095-default-flexpsql   Perform Point-in-Time Restore   The restore can be performed in the Azure Portal or using the Azure CLI. In the Azure Portal select the Resource Group of your Viya Deployment. Under the listed resources, select the Azure PostGresSQL Flexible server. Within the flexible server under Settings on the right, select Backup and Restore. Follow these steps to restore:   Step 1. Select the Fast Restore option for the Automated Backup with the Completion timestamp before, but closest to the restore point.     Step 2. Select the radio button Select a custom restore point and provide.   The date and time of the restore point that matches the backup ID. Give the new database instance a unique name.     Select Review and Create and then Create.   The deployment of the new instance will start with the message "Deployment in Progress". When it is completed the UI will output the message "Your deployment is complete".  This will complete the restoration of the Viya Infrastructure data server. It is restored to the new instance name and in Azure you now have two instances, the Viya environment still points to the original instance. In the following steps of the restore, we will switch Viya to point to the restored instance.   Restore the SAS Viya Backup   When using PITR for the PostGresSQL database you must select a backup package that does not include the SAS Viya Infrastructure Data Server data. The restore process happens in two steps. These steps are summarized here and covered in more detail in my previous post.   Step 1 The Restore Job   Restores the SAS Configuration Server Stop the CAS Server(s)   Step 2 Clear the CAS PV's and restart CAS in RESTORE mode   Delete existing data from the CAS PV's Restores CAS data and content   Run the Restore Job   Firstly, identify the sas-restore-job-parameters configMap that needs to be modified. This step returns the config map for the restore job.   restore_config_map=$(kubectl describe cronjob sas-restore-job | grep -i sas-restore-job-parameters | awk '{print $1}'|head -n 1) echo The current restore Config Map is: $restore_config_map   Output:   The current restore Config Map is: sas-restore-job-parameters-hgd4ftbmmm   Edit the configMap to set the restore parameters.   SAS_BACKUP_ID to the backup ID of the package to restore. SAS_DEPLOYMENT_START_MODE to RESTORE. AUTO_SWITCH_POSTGRES_HOST to True. DATASERVER_HOST_MAP to map the name of the restored PostGresSQL Server.   We have stored these values in environment variables in prior steps. This code updates the restore configMap with the relevent values and checks the values have been updated.   RESTORE_DB_HOST=${RESTORE_DB_NAME}.postgres.database.azure.com kubectl patch cm $restore_config_map --type json -p '[ {"op": "replace", "path": "/data/SAS_BACKUP_ID", "value":"'${backupid}'"},{"op": "replace", "path": "/data/SAS_LOG_LEVEL", "value":"DEBUG"},{"op": "replace", "path": "/data/SAS_DEPLOYMENT_START_MODE", "value":"RESTORE"},{"op": "replace", "path": "/data/AUTO_SWITCH_POSTGRES_HOST", "value":"True" },{"op": "replace", "path": "/data/DATASERVER_HOST_MAP", "value":"sas-platform-postgres:'${RESTORE_DB_HOST}'"}]' kubectl describe cm $restore_config_map     Start the Restore Job from the Restore cronjob. This process will restore the SAS Configuration Server. In addition, it will stop the CAS server to prepare for restore of the CAS server.   kubectl create job --from=cronjob/sas-restore-job sas-restore-job   After giving the job some time to run you can check for specific messages in the log to check the status.   kubectl logs -l "job-name=sas-restore-job" -c sas-restore-job --tail 1000 | klog | grep "restore job completed successfully" -B 3 -A 10   IMAGE   Restore the CAS Server   With the restore job completed the second step will restore the CAS server. This process is covered at a high level here but is covered in more detail in this post.   The restore job has stopped all CAS Servers in the environment. To restore CAS, the CAS Server will be started in RESTORE mode and data and configuration will be restored during server startup. Firstly, let’s check that CAS is not running.   kubectl get pods --selector="app.kubernetes.io/managed-by==sas-cas-operator"   Output:   No Resources in yournamespace namespace.   The process uses two provided CAS scripts.   sas-backup-pv-copy-cleanup.sh deletes the existing data from the CAS PV's. scale-up-cas.sh starts the CAS server(s) in RESTORE mode.   Make the provided CAS scripts executable.       Restoring the data to the two CAS file PVCs requires a clean volume. Run the sas-backup-pv-copy-cleanup.sh script to clean up the CAS PVCs. This step deletes the existing data on the CAS permstore(cas-default-permstore) and CAS data(cas-default-data) PVCs.   ~/project/deploy/${current_namespace}/sas-bases/examples/restore/scripts/sas-backup-pv-copy-cleanup.sh gelcorp remove "default"   Startup the CAS Server   ~/project/deploy/${current_namespace}/sas-bases/examples/restore/scripts/scale-up-cas.sh gelcorp "default"   When the CAS server starts it checks the Restore Job configMap attribute SAS_DEPLOYMENT_START_MODE. If it is set to RESTORE, the CAS server will start and restore the data from the directory that matches the BACKUP_ID in the sas-cas-backup-data PVC.   Check the log to see if the CAS server performed the restore. The logs should show the start of the restore process that restores the backup content to the target CAS persistent volumes.   kubectl logs sas-cas-server-default-controller -c sas-cas-server | gel_log | grep -A 10 "RESTORE"   Finalize the Restore   To complete the switch over to the newly restored Postgres instance, restart all the services with the sas-stop-all and sas-start-all cronjobs. This code runs the stop job and waits for all the PODS on the namespace to stop before running the start and job and then waiting for Viya to start.   kubectl create job --from cronjobs/sas-stop-all restart-stop RUNNING_PODS=$(kubectl get pods --field-selector=status.phase=Running -o go-template='{{len .items}}') until [ $RUNNING_PODS -lt 2 ]; do echo Running Pods: $RUNNING_PODS; RUNNING_PODS=$(kubectl get pods --field-selector=status.phase=Running -o go-template='{{len .items}}'); done kubectl create job --from cronjobs/sas-start-all restart-start   To ensure that SAS Viya 4 is referencing the restored PostgreSQL instance, execute the following command for each PostgresSQL instance in the cluster. The command should show the address and services as the new name that you gave to the PostgresSQL database.   HTTP_PROTOCOL=https CONSUL_SVC_NAME=sas-postgres kubectl exec -it sas-consul-server-0 -c sas-consul-server -- bash -c "export CONSUL_HTTP_ADDR=$HTTP_PROTOCOL://localhost:8500; /opt/sas/viya/home/bin/sas-bootstrap-config catalog service $CONSUL_SVC_NAME" { "items": [ { "address": "d20240115-134651f-blitz-p02095-default-flexpsql.postgres.database.azure.com", "createIndex": 692506, "datacenter": "viya", "modifyIndex": 692506, "node": "sas-postgres", "nodeMeta": null, "serviceAddress": "d0240115-134651f-blitz-p02095-default-flexpsql.postgres.database.azure.com", "serviceEnableTagOverride": false, ........   Finally, we will reset the SAS restore job configMap parameters.   kubectl patch cm $restore_config_map --type json -p '[{ "op": "remove", "path": "/data/SAS_BACKUP_ID" },{"op": "remove", "path": "/data/SAS_DEPLOYMENT_START_MODE"} ,{"op": "remove", "path": "/data/AUTO_SWITCH_POSTGRES_HOST" },{"op": "remove", "path": "/data/DATASERVER_HOST_MAP"} ]'   The Viya manifests point to the PostgreSQL server used before the restore. An important final step ensures that when any future changes are made Viya continues to point to the restored server.  To ensure that Viya continues to point to the restored server update the manifests to point to the new host for the PostgreSQL server. The file to be updated is usually at $deploy/site-config/postgres /{ POSTGRES-SERVER-NAME }-dataserver-transformer.yaml. Here is an example of an updated file.   apiVersion: builtin kind: PatchTransformer metadata: name: sas-platform-postgres-dataserver-transformer patch: |- - op: replace path: /spec/users/0/credentials/input value: secretRef: name: platform-db-user usernameKey: username # For internal use, do not modify passwordKey: password # For internal use, do not modify - op: replace path: /spec/registrations/0/host value: d20240115-134651f-boxer-p02095-default-flexpsql - op: replace path: /spec/registrations/0/port value: 5432 - op: replace path: /spec/ssl value: true target: group: webinfdsvr.sas.com kind: DataServer version: v1beta1 name: sas-platform-postgres   Considerations   The Azure Flexible server for Postgres has a default retention period of 7 days. The retention period for the Postgres should be changed so that it is in synch with the SAS Viya backups. SAS Viya backups have a retention period of 30 days by default. Matching retention periods will ensure that if we need to restore a backup the required PostgreSQL and SAS data is available.   The restore process created a new PostGresSQL instance. You may need to keep the old and new instances around for a while to provide the ability to do a PITR during the 30-day retention period.   Wrap-Up   The addition of support for using third-party functionality to backup and restore the SAS Viya Infrastructure Data Server portion of the Viya deployment is a welcome update. This support depends on the database supporting Point-in-time restore. This addition adds flexibility to the SAS Viya Backup and Restore functionality and will be particularly useful when the volume of data stored in the Infrastructure Data Server is large. The SAS Viya Administration Backup and Restore documentation has been updated to cover the new workflow.     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya stable 2023.10 introduces a simplified approach to restoring a Viya backup. In prior releases, an administrator was required to build and apply temporary manifests to initiate a restore. In 2023.10 (and forward) this is no longer the case. In this post, I will review the new and improved functionality.   Viya Backup and Restore Review   In a previous post, I provided an overview of SAS Viya 4 Backup and Restore. To summarize, a Viya backup includes:​   ​Content stored in the Infrastructure Data Server​ (reports, folders, rules, etc.) Configuration stored in the SAS Configuration Server​ (application configuration settings etc.) CAS permstore ​(Caslib definitions, access controls, etc.) Default CAS persistent volume content ​(contents of the Public caslib etc.)   Viya backup and restore are implemented using native Kubernetes functionality. Backup and restore are implemented as Kubernetes jobs and cronJobs. The restore process has two high-level steps.   Step 1: The Restore Job restores the SAS Infrastructure Data Server and Configuration Server Step 2: The CAS Server starts in RESTORE mode to restore CAS configuration and data.   What has Changed?   So what has changed? In 2023.10 the restore process performs the same two steps, however, the method of initiating a restore has improved. Until 2023.10 the restore process involved building and applying temporary Kubernetes manifests to perform the two steps; running the Restore Job and restarting CAS in RESTORE mode. After restoring the backup was complete the manifests that manage the deployment had to be reset to their original state and the temporary manifests discarded. The restore process was not particularly easy for the Viya administrator.  The new process does not require the creation of temporary manifests. Let's see how it works now.   Step 1 Running the Restore Job   Each individual backup is identified using a timestamp value called the backup ID. In the command below, we retrieve the backup ID of the backup to restore from the ad-hoc backup job that created the backup.   backupid=$(yq4 eval '(.metadata.labels."sas.com/sas-backup-id")' <(kubectl get job sas-scheduled-backup-job-adhoc-001 -o yaml)) echo ${backupid}   Output:   2023-11-09T15_32_28_628_0700   When selecting a backup to restore you should always check the status.json in the sas-common-backup-data PVC directory to be sure that the backup you are restoring was completed successfully. The status.json contains detailed information about the backup. For a successful backup, the file should have the value: sas.com/sas-backup-job-status: Completed.   The sas-restore-job-parameters configMap is how we pass settings to the restore process.  The configMap will be referenced by the restore job and by the CAS Server when it starts. The two parameters to set for a restore are:   SAS_BACKUP_ID: the backup ID of the backup to restore. For example, 2023-11-09T15_32_28_628_0700 SAS_DEPLOYMENT_START_MODE: RESTORE   The patch command patches the configMap updating the values of SAS_BACKUP_ID and SAS_DEPLOYMENT_START_MODE:   restore_config_map=$(kubectl describe cronjob sas-restore-job | grep -i sas-restore-job-parameters | awk '{print $1}'|head -n 1) echo The current restore Config Map is: $restore_config_map kubectl patch cm $restore_config_map --type json -p '[ {"op": "replace", "path": "/data/SAS_BACKUP_ID", "value":"'${backupid}'"}, {"op": "replace", "path": "/data/SAS_DEPLOYMENT_START_MODE", "value":"RESTORE" }]'   Output:   The current restore Config Map is: sas-restore-job-parameters-bm48bd82bg configmap/sas-restore-job-parameters-bm48bd82bg patched   Using the following command we can view the updated configMap and make sure that the SAS_BACKUP_ID and SAS_DEPLOYMENT_START_MODE parameters are correctly set.   kubectl describe cm $restore_config_map   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   With the parameters set correctly, we can now start the Restore Job from the Restore cronJob. This process will restore the SAS Infrastructure Data Server and the SAS Configuration Server. In addition, it will stop the CAS server which is a pre-requisite for the second step which will restore the CAS server.   kubectl create job --from=cronjob/sas-restore-job sas-restore-job   Output:   job.batch/sas-restore-job created   You can view the job log as it runs with this command, using the -f parameter to stream the log to the screen.   kubectl logs -l "job-name=sas-restore-job" -f -c sas-restore-job | gel_log     Note: in the previous command we pipe to a custom function(gel_log) to reformat the log from JSON to a more human-readable format. The function is shown below for your information.   gel_log () { jq -R -r '. as $line | try (fromjson| "\(.level | ascii_upcase) \(.timeStamp) '['\(.source)']-' \(.message) " ) catch $line ' }   You can check the status of a restore job with this command. The status should eventually change from "Running" to "Completed".   kubectl get jobs -l "sas.com/backup-job-type=restore" -L "sas.com/sas-backup-id,sas.com/backup-job-type,sas.com/sas-restore-status"   Output:   NAME COMPLETIONS DURATION AGE SAS-BACKUP-ID BACKUP-JOB-TYPE SAS-RESTORE-STATUS sas-restore-job 0/1 42s 42s 2023-11-16T19_48_28_628_0700 restore Running   To make sure the job has run successfully check the log for the message "restore job completed successfully."   kubectl logs -l "job-name=sas-restore-job" -c sas-restore-job --tail 1000 | gel_log | grep "restore job completed successfully" -B 3 -A 1     The restore job will perform a rolling start of many of the SAS Viya Pods. Before moving on to the next step we should check that two of the key PODS (SAS logon and Configuration) are up and running.   kubectl get pods -l app=sas-logon-app kubectl get pods -l app=sas-configuration   Output:   [cloud-user@pdcesx11133 from35]$ kubectl get pods -l app=sas-logon-app NAME READY STATUS RESTARTS AGE sas-logon-app-5dcb9df44d-hb8gd 1/1 Running 0 2m26s [cloud-user@pdcesx11133 from35]$ kubectl get pods -l app=sas-configuration NAME READY STATUS RESTARTS AGE sas-configuration-68b558b8b9-g7dg4 1/1 Running 0 2m46s   Step 2 Restore the CAS Server   With the restore job completed the second step will restore the CAS server. The restore job has stopped all CAS Servers in the environment. To restore CAS, the CAS Server will be started in RESTORE mode and data and configuration will be restored during server startup. Firstly, let's check that CAS is not running.   kubectl get pods --selector="casoperator.sas.com/server==default" -n gelcorp   Expected Output:   No resources found in gelcorp namespace.   To replace the old manifest approach two new scripts are now used to initiate the CAS restore: The scripts are delivered with the deployment assets in the directory sas-bases/examples/restore/scripts. The two scripts are:   sas-backup-pv-copy-cleanup.sh deletes the existing data from the CAS persistent volumes. scale-up-cas.sh starts the CAS server(s).   The scripts are delivered with the deployment assets in the directory sas-bases/examples/restore/scripts. To run the scripts, we need to make them executable.   chmod +x ~/project/deploy/${current_namespace}/sas-bases/examples/restore/scripts/*.sh   The restoration of the data to the two CAS file PVCs requires a clean volume. Run the sas-backup-pv-copy-cleanup.sh script to clean up the CAS PVCs. This step deletes the existing data on the CAS permstore(cas-default-permstore)  and CAS data(cas-default-data) PVCs. The parameters of the script are:   namespace operation tenant list (default for non-multi-tenant)   cd ~/project/deploy/${current_namespace}/sas-bases/examples/restore/scripts/ ./sas-backup-pv-copy-cleanup.sh gelcorp remove "default"   Output:   The cleanup pods are created, and they are in a running state. Ensure that all pods are completed. To check the status of the cleanup pods, run the following command. kubectl -n gelcorp get pods -l sas.com/backup-job-type=sas-backup-pv-copy-cleanup | grep 21bef2c   The script creates a Kubernetes Job that clears the key data from the CAS PVCs so that it can be restored from the backup package. Using the command provided in the script output we can view the status of the job and for more details we can view the log of the job.   kubectl -n gelcorp get pods -l sas.com/backup-job-type=sas-backup-pv-copy-cleanup | grep 21bef2c kubectl -n gelcorp logs -l sas.com/backup-job-type=sas-backup-pv-copy-cleanup     With the CAS PVCs successfully cleaned, we can start up the CAS server(s) using scale-up-cas.sh. The parameters of the script are:   namespace tenant list (default for non-multi-tenant)   cd ~/project/deploy/${current_namespace}/sas-bases/examples/restore/scripts/ ./scale-up-cas.sh gelcorp "default"   casdeployment.viya.sas.com/default patched   When the CAS server starts it checks the Restore Job configMap attribute SAS_DEPLOYMENT_START_MODE.  If it is set to RESTORE, the CAS server will start and restore the data from the directory that matches the BACKUP_ID in the sas-cas-backup-data PVC.   You can check the CAS Server log to see if the restore was performed. The logs will show the start of the restore process and details of the restore of the backup content from the backup package to the target CAS persistent volumes.   kubectl logs sas-cas-server-default-controller -c sas-cas-server | grep -A 10 "RESTORE"   An important final step is to reset all SAS restore job configMap parameters. If we don't perform this step the CAS server will attempt to restore the backup from the package on every restart.   kubectl patch cm $restore_config_map --type json -p '[{ "op": "remove", "path": "/data/SAS_BACKUP_ID" },{"op": "remove", "path": "/data/SAS_DEPLOYMENT_START_MODE"}]'   Wrap Up   Restore of a SAS Viya backup is now initiated using kubectl commands and scripts. The new and improved restore process is currently supported for Backup and Restore and Viya 4 to Viya 4 Migration.  Currently, the old method of building and applying manifests is also supported. There are plans to add support for the new method for Viya 3. x to 4 Migration. I hope you found this useful. In the Backup and Restore area look for more exciting updates and related blog posts in the coming months.   The new restore process is documented:   SAS Viya Platform Administration Backup and Restore: Perform a Restore (2023.11) SAS Viya Platform Administration Migration: Moving from SAS Viya 4: Restore(2023.11)     Find more articles from SAS Global Enablement and Learning here. ... View more

A SAS Model Studio project is a top-level container for analytic work in Model Studio. A Model Studio project contains the data source, the pipelines, and related project metadata (such as project type, project creator, share list, and last update history). SAS Model Studio projects are, by default, saved in Viya Content to the user’s “My Folder”. Content Migration of Viya Model Studio Projects has been troublesome in some cases because of the strict prerequisites. Now we have improved functionality that makes life easier. In this post, I will examine these improvements.   Firstly, what are the prerequisites? To import a Model Studio Project:   The user who owns the project must exist and must have logged on in the target environment​​ Before import, data must be available and loaded into the memory in the target environment (in the same caslib as in the source environment). A user-created pipeline or node template must be promoted separately.   For more details on these items see the documentation here.   These prerequisites, particularly the requirement that the user that owns the project must already exist and have logged in, have at times made it challenging to migrate Model Studio projects.  This requirement makes it difficult to migrate Model Studio Projects when the source owner: is no longer at the organization. doesn’t have authorization to the target. is a different user in the target. is the same in the target, but the user ID is managed differently. For example, on the development server, the id is dev_user1 and on the test server, the id is test_user1.   One solution to the problem is to change the ownership of a Model Studio Project to a user who does exist and has logged on to the target environment. In the past, there was no supported way to change ownership of a project. New functionality in the transfer plugin of the sas-viya CLI now allows an administrator to change the ownership of Model Studio Projects and other content types prior to import.   New functionality in sas-viya transfer   There are two methods to migrate Model Studio projects between environments you can export using: The transfer service using SAS Environment Manager or the sas-viya CLI, this creates a transfer package (JSON file)​ SAS Model Studio creates a ZIP file containing one JSON transfer file per project   The new method of changing ownership works with the transfer method of migrating. The process works like this: Use the transfer plug-ins list-owners command to generate an "owners list" JSON file from the transfer package Modify the generated file to map the old owner to a new owner Use the transfer plug-in replace-owners command and the edited "owners list" file's content to create a new transfer package file with the ownership of the content in the package changed.   NOTE: to be clear the ownership of the object is changed in the transfer package not in the source system.   Changing Ownership of Model Studio Projects for Import to Viya   Let’s look at the process in detail using an example. In this example, Sarah has some Model Studio projects in one Viya environment and she needs to share them with a different user in another environment.   She starts by using SAS Environment Manager to export a transfer package and shares that transfer package with the other user GEL Administrator (geladm).   Review the Transfer Package   Firstly, let’s review the content of the provided transfer package.   packageid=$(/opt/sas/viya/home/bin/sas-viya --output json transfer upload --file /tmp/viyapackages/SarahMyFolder.json | jq | jq -r '.["id"]') echo "View the package with id" $packageid sas-viya --output text transfer show --details --id $packageid   The output shows that the package contains the folder path to Sarah’s “My Folder” and the content of the folder which consists of the Model Studio Project and supporting resources.       Generate the  "owners list" file   The first step is to generate the "owners list" file from the transfer package. To do that we can use the new “list-owners” command pass it the transfer package file and output the results to an owners list file. The owner's list file is in JSON format and contains a map of the existing owner(sourceUserID) to a potential new owner(targetUSERID). The source and target are initially the same.     sas-viya transfer list-owners --source-file "/mnt/workshop_files/SarahModelStudio/SarahMyFolder.json" --owners-list-file /tmp/theowners.json [ { "sourceUserID": "Sarah", "targetUserID": "Sarah" }, { "sourceUserID": "anonymousUser", "targetUserID": "anonymousUser" } ]   Update the "owners list" file and change ownership   To change the ownership of objects in this package edit the file and change the targetUserID to the user in the target system who you want to own the content.     [ { "sourceUserID": "Sarah", "targetUserID": "geladm" }, { "sourceUserID": "anonymousUser", "targetUserID": "anonymousUser" } ]   Using the updated file create a new transfer package with the ownership changed from the user Sarah to the user geladm. Pass to the transfer replace-owners command: --source-file the source transfer package --owners-list-file the update owners list file --target-file a file to store the updated transfer package   sas-viya transfer replace-owners --source-file /mnt/workshop_files/SarahModelStudio/SarahMyFolder.json --owners-list-file /tmp/theowner-geladm.json --target-file /tmp/geladm-modelstudio.json     Notice the output indicates that the ownership in the package has been changed based on the content of the file. Simple as that. Of course, to truly test if this works we need to import the package.   Import the new transfer package file   In SAS Environment Manager import the package (you could also use the CLI). When we open the package in the Environment Manager import interface notice that the content will be imported to the user geladm’s personal folder. This is the first clue that we have successfully changed the ownership of the content.         After walking through the import wizard the summary screen shows the content, including the Model Studio projects was imported successfully into the My Folder area of the geladm user.     In the Content Area of SAS Environment Manager, we can see the content, originally owned by Sarah in the source environment is now owned by geladm and is located in geladm's "My Folder".         Wrap Up   When migrating Model Studio projects there are specific prerequisites required to ensure a successful migration. One of those requirements is that the user that owns the project must already exist and have logged in to the target system. This has at times made it challenging to migrate Model Studio projects. One solution to the problem is to change the ownership of a Model Studio Project to a user who does exist and has logged on to the target environment. New functionality in the transfer plugin of the sas-viya CLI now allows an administrator to change the ownership of Model Studio Projects and other content types prior to import.   For more information:   Content Migration Considerations for SAS Visual Data Mining and Machine Learning   Machine Learning Users Guide: Importing and Exporting Projects   ... View more

In previous posts published in the early days of SAS Viya on Kubernetes, I introduced the benefits of running the SAS Viya CLI inside a docker container and discussed how you can orchestrate Viya administration tasks using a CLI container and Argo Workflow. You can catch up on updated versions of these two posts here:   SAS administration command-line interface in a container: Part 1 Docker SAS administration command-line interface in a container: Part 2 Kubernetes and Argo Workflow   At SAS recent developments around orchestration and scheduling have focused on using Apache Airflow to schedule and manage flows. The SAS Airflow Provider allows SAS Viya administrators/power users to orchestrate SAS jobs and SAS Studio flows using Apache Airflow. Argo Workflow allowed us to orchestrate common administration tasks performed with the SAS Viya CLI. With the focus now on Apache Airflow I wondered if it could do the same. The short answer is yes, read on for the details.   Apache Airflow v Argo Workflow   Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post.   Firstly let's compare the two applications. Apache's Airflow and Argo Workflows are two of the most popular workflow engines available. A workflow engine is a platform for scheduling, starting, stopping, organizing, and monitoring flows containing sets of related tasks. How do Apache airflow and Argo workflow compare? First, the similarities. They both use Directed Acyclic Graph (DAG). A DAG models multi-step workflows as a sequence of tasks and includes the dependencies between tasks. They can both be used in Kuberenetes, Argo is entirely Kuberenetes native while Airflow is Kubernetes-friendly, and can be run in a Kubernetes cluster to take advantage of the increased stability and autoscaling options that Kubernetes provides. The big difference between the two is how you define your DAG workflows, in Argo you define them in YAML whereas in Airflow you define them in Python. (You have to say that Argo does have the cooler logo :))   Apache Airflow and the SAS Viya CLI   Apache Airflow is modular and can interface with other software by installing additional packages, called providers or operators. The SAS Airflow Provider is a good example of this, it was created to allow Airflow to run SAS processing. Other commonly used operators include, for example, the Bash operator and the Python operator. The Bash operator can run shell scripts or a set of commands and would be one option for running the sas-viya CLI commands. You can see an example of this in Michael’s post here. However, the obvious choice for us is the Airflow KubernetesPodOperator which executes a task defined as a Docker image in a Kubernetes Pod. Let’s look at how we run our sas-viya administration workflow with Airflow and the KubernetesPodOperator.   In our environment, Apache Airflow is installed and configured in Kuberenetes as described in Nicolas's post here. One of the key aspects of integration is that the environment is configured so that Airflow DAGs (Python scripts) are automatically discovered by the Airflow framework when they are saved in a specific directory.   Define and Run an Administrative Workflow   Here is how we define and run a Viya administration workflow in Airflow. Our flow will automate the performance of the initial administration tasks for a new Viya environment. It will:   Setup identities Create a folder structure Apply an authorization schema to the folder structure Create Caslibs for data access Load Data to CAS Apply CAS authorization Load Viya Content Validate that the previous steps were successful   The first thing we need is a container image that includes the CLI and any dependent software. How we achieve this was covered in this post. Next, we define our Python script, which will create the flow. Let’s take a walk through the script.   At the start, we import the Python Packages we will need in the script. Then define the Volumes, Volume mounts in a Python list, and the environment variables in a Python dictionary. These elements will be passed to the KubernetesPodOperator. The ConfigMap volumes make the CLI configuration and credentials file and the Viya environment certificates available to each POD. The environment variables define the CLI profile to use and the paths to the certificate files. When the POD for each task is started the environment variables will be set and the volumes mounted.     Next, we create the DAG giving it a name (01-load-content-flow) and setting default attributes including the schedule, start date, etc. In our example, we are going to create a flow that is not scheduled and we will trigger it manually in the UI. If you want to provide a schedule you provide it in Cron format.     Following the DAG we define the tasks. Each task in our case uses the KubernetesPodOperator and will run a script that contains SAS Viya CLI commands. For each task, we provide the container image, give the task a name, and reference the volume, volume mount, and environment variables. We pass the bash -c command with a single argument which is the script that will run. Each script contains CLI commands that achieve the result of the task.     The final part of the Python program defines the dependencies for the tasks in the DAG.     Save the Python program to our directory that Airflow is configured to monitor and the DAG is imported to the Apache Airflow and we can view it in the application. Notice the dependencies setup in the DAG definition are honored.  As was mentioned before the flow can be scheduled or triggered from the UI.     The application also allows you to monitor flows.  The screenshot below shows a flow in progress, the color-coding shows that 5 tasks have been completed successfully, one task is running and the final two tasks are waiting to start.     When tasks are running or complete you can view the logs. In the screenshot below from the data load task we can see messages indicating that the data is being loaded to CAS.     You can also use tools like kubectl or Lens to monitor the process.  A kubectl get pods shows a completed POD for each task in the flow (pods can also be automatically deleted on completion).   kubectl get pods -n airflow | grep task     Wrap-Up   In this post we have looked at two Automation and Workflow Management tools; Apache Airflow and Argo Workflow. Both applications provide similar functionality. The main difference between the two applications is the language used to define flows. Apache Airflow uses Python and Argo Workflow uses YAML. Apache Airflow can be run in and take advantage of Kubernetes. In a prior post, we automated the execution of Viya administration tasks with Argo Workflow. In this post, we demonstrated that you can automate, schedule, execute, and monitor SAS Viya administration tasks using Apache Airflow, a containerized SAS Viya CLI and the Airflow KubernetesPODOperator.     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS 9 Content Assessment is one of the key tools used in planning a migration from SAS 9.4 to the SAS Viya Platform. The most recent release of Content Assessment includes significant improvements in the Profile for SAS Data Integration Jobs report and some changes in the terminology used to understand the readiness status of Data Integration jobs and transformations. In this post, I will review and explain these changes.  The SAS Profile Content Assessment application provides detailed information on the attributes of SAS 9 content. Profile delivers a series of reports for each SAS 9.4 content type.   The reports include details like:   Tasks used in Enterprise Guide projects Transformations used in Data Integration Studio Jobs Calculations used in cubes and information maps.   The Content Assessment Profile application and reports help you to understand the details of SAS 9 Content and how those details impact migration to SAS Viya. The SAS Profile for Data Integration Jobs report is the main report that covers SAS Data Integration Studio Jobs. SAS Data Integration studio jobs are made up of transformations (the nodes in the jobs). Jobs migrate to SAS Studio flows which include steps (the nodes in the Studio flows).   Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post.   The migration process will attempt to migrate a transformation to an equivalent step or steps in Studio. If a transformation cannot be migrated to equivalent step(s), it is often migrated to a Studio Program step that includes SAS code to perform the Transformations function. A transformation may be prevented from migrating to an equivalent step because there is no matching step in Studio, or some attribute of the transformations is not yet supported. The SAS 9 Profile for Data Integration report helps you to understand what may happen to the Data Integration Studio Jobs and their Transformations during the migration process from SAS 9 to Viya.   Terminology   Let’s look at the new terminology around the status of Data integration Studio jobs and transformations.   Transformations   Transformations are classified as Ideal Fit, Code Fit, No Fit, or Undetermined Fit.   A transformation is considered:   Ideal Fit if the transformation migration is automatic and it migrates to equivalent step(s) in SAS Studio. Code Fit if the transformation migration is automatic and it migrates to a Program step in SAS Studio. No Fit if the transformation does not currently have a planned feature replacement in SAS Viya. Undetermined Fit if the transformation was unable to be profiled or evaluated for migration status.   Transformations within a Data Integration job that are profiled will fall into one of these categories.   Jobs   In the case of jobs, we can measure the fit of a job by looking at the fit of its constituent transformations. A simple formula can be used to calculate the Ideal Fit Percentage of a job. The ideal fit percentage is calculated by dividing the number of transformations in the job that are ideal fit by the total transformations in the job to get the ideal fit percentage.   ideal fit percentage = (the number of transformations in a job that are ideal fit / by the total number of transformations in the job) *100   The Percentage of Transforms with Ideal Fit shows how much of a job will migrate to equivalent objects in SAS Viya. Jobs with a higher percentage will take advantage of the equivalent steps in Studio, jobs with a lower percentage should still migrate, but some of the functionality in Studio will be provided by code within Program steps.   The Report   Overview   Let’s look at the report to see the new terminology in action.   The Overview tab in the report is a summary of the status of all the Jobs and Transformations in the deployment. You can see in this deployment there are 22 jobs with 46 Transformations. Of the 22 jobs, all are “Migration Candidates”. A job is considered a migration candidate when all of its transformations can be migrated (either ideal fit or code fit).       Transformation Migration Summary   It can be useful to start with the overview of the transformation usage in the deployment provided by the Transform Migration Summary tab.  From this page, we can see the greatest success and common issues in the assessed environment. The table on the left of the page shows the most frequently used SAS 9 transformations in the SAS 9 environment and the Steps they become in SAS Viya. In this report, the File Reader was the most frequently used transformation and we can see that it is migrated to a Program Step. The table to the right shows the most common issues that will cause transformations to convert to a Program Step.     In this case, the issues are related to a mismatch between the definition of columns in the source and target tables. With the issue identified, changes could be made to the jobs to make them a better fit for migration to SAS Studio. In addition, customers can share these results with SAS resulting in a potentially improved migration experience for this and other common scenarios.   Transformation Migration Details   The Transformation Migration Details tab allows you to further explore the detail of the migration of transformations. It also allows the selection of individual jobs, and the exploration of what will happen to the specific transformation in the jobs. Let us look at a couple of jobs.   Ideal Fit Job: 01_Split_Finance_Data   Here on the Transformation Migration Details tab, we have selected 01_Split_Finance_Data. The job has a 100%  of its transformations ideal fit meaning that all transformations migrate to an equivalent step in Studio. The report shows the:   File Writer Transformations will migrate to an Export Studio step Extract Transformation will migrate to a Query step Splitter Transformation will migrate to a Branch Rows step.     Code Fit Job: DIFT Profit   Let’s look at another job that was classified as code fit and use the report to understand why. The Job DIFT Profit has one transformation that is Ideal Fit and two are Code Fit.     The report shows the :   Extract transformation will migrate to a Query step File Reader will migrate to a Program step Table Loader will migrate to a Program step. The report provides the conversion message explaining why the transformation will migrate to a program step. Focusing on the Table Loader transformation (which could potentially migrate to a Load Table Studio step) we can further investigate why it will migrate to a Program step. The conversion message notes at the bottom of the screen indicate that the Table Loader will migrate to a Program step because the Append uses the FORCE option.     Selecting a File Reader transformation, the conversion message indicates there is an issue with a mismatch of the SAS  informat for columns in the source and target table. Selecting a row in the table will filter the lower table where object-specific details are provided. In this case, the object-specific details table identifies the table and column where the problem is occurring.       Job Migration   The Job Migration tab provides a status for each Job that has been profiled. The tab also provides links to help explain the report and the terminology used. The Job listing in this tab can be downloaded to a CSV file that will drive the migration process. The CSV file is used as an input to the CreateSASPackages application which will automate the export of the Data Integration Studio jobs that you select for migration. You can use the slider to investigate the impact of lowering the threshold for the "percentage of transforms with ideal fit".  When you adjust the slider Jobs higher than the ideal fit threshold are color-coded blue in the table listing. The buttons can be used to filter the jobs in the table, and the checkboxes to select group(s) of jobs to export for package creation.     Summary   I hope this summary of the updates to the SAS 9 Profile for Data Integration Studio report and transformations has been useful.  The report allows you to explore the migration status of the Data Integration Jobs and Transformations in your SAS 9 environment. The new terminology and additional functionality in the SAS 9 Profile for Data Integration report will be very helpful in planning for the migration of Data Integration Studio Jobs to the SAS Viya platform. I would like to thank Millie Lombardo and Scott McCauley for their input on this post. For more information:   Content Assessment Documentation SAS DEMO | Migrating SAS Data Integration Studio Jobs to SAS Studio Flows Content Assessment delivers utility applications to support SAS Viya Content Migration     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS 9 Content Assessment is a very popular way to examine and document the content of a SAS 9.4 environment. Did you know you have similar functionality in SAS Viya? In this blog post, I will look at the inventory options in SAS Viya. In Viya 4 recent improvements have made scanning your environment and examining the configuration and content even easier. It is a recommended best practice to run a scan before and after any major changes in your Viya environment. For example, scan before and after a system migration, an update of the Viya software, before a backup, and after a restore, and before and after migrating content.   In this post, we will focus on Viya 4, but first, let us recap what is available on Viya 3.5.   SAS Viya 3.5 Inventory scan and reporting   In SAS Viya 3.5 the scanning functionality that collects the inventory data is provided by the inventory plug-in of the sas-admin -line interface. The inventory plug-in of the CLI: Generates the Plan and Backup Ansible playbook​ Delivers the Viya Inventory Reports​ The Plan and Backup playbook is the way you run a scan, the playbook creates and publishes all the data that supports the Viya Inventory reports. I covered this process in the first part of this YouTube video. Recent updates to the playbook have added support for scanning a Viya 3.x multi-tenant environment.   SAS Viya 4 Inventory scan and reporting   In Viya 2022.09 and later inventory scanning in Viya 4 became easier. A Cronjob the sas-viya-inventory-collector makes it simple to scan and inventory the Viya 4 environment. The Cronjob also supports the ability to compare the two most recent scans.  Additionally then Cronjob provides support. for scanning deployments that are multi-tenant and/or SCIM-enabled.   The Inventory Reports are delivered when Viya is installed, there are two reports available at Products > SAS Environment Manager > Dashboard   SAS Viya 4 Inventory: surfaces the result of the most recent inventory scan of the Viya environment. SAS Viya Inventory Comparison: compares the last two inventory scan results. The reports are supported by data loaded into the SystemData CAS library by the scan process. Two tables hold the results of the most recent inventory scan.   SASVIYAINVENTORY4  SASVIYAINVENTORY4_CASSVRDETAILS   To enable comparison after the second scan runs each scan also keeps a copy of the prior SASVIYAINVENTORY4 table. If you have run at least two scans you should also have the table SASVIYAINVENTORY4_PRIOR.   Running a Viya 4 Inventory scan   Scan functionality is delivered by the sas-viya-inventory-collector Kubernetes Cronjob. The Cronjob, by default, is not scheduled. Instead, it is delivered as a convenient way of performing a scan. The following command runs a scan by creating and running a Kubernetes job from the Cronjob. The new job is named sas-inventory-collector-job (you can give it any name).   kubectl create job --from=cronjob/sas-inventory-collector sas-inventory-collector-job   To follow the log as the job runs execute:   kubectl logs -l job-name=sas-inventory-collector-job -f | klog Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The example scan will scan a non-multi-tenant environment. In a multi-tenant environment an inventory can be performed on a per-tenant basis by setting the tenant environment variable to the tenant name prior to running a scan.   kubectl set env cronjob/sas-inventory-collector TENANT=tenant_name   When the job is complete you can view the SAS Viya Inventory Report (more on the report later).   Running an Inventory Comparison   If you have run at least two scans you can do a comparison. The comparison scan will compare the most recent scan results to the previous scan results. To perform a comparison set the COMPARISON environment variable to true and create a job (When the job is completed it is a good practice to reset the environment variable.)   kubectl set env cronjob/sas-inventory-collector COMPARISON=true kubectl create job --from=cronjob/sas-inventory-collector sas-inventory-comparison-job kubectl set env cronjob/sas-inventory-collector COMPARISON-   Follow the job log:   kubectl logs -l job-name=sas-inventory-collector-comparison -f | klog   The output of the comparison scan process looks like this.     Comparison with Viya 3.5 after Migration   In a migration from Viya 3.x you can compare your new Viya 4 environment to the source Viya 3.x environment. In order to do this Before migration perform an inventory scan of the source environment (for details see this YouTube video.) Perform a full-system migration of the Viya  3.5 environment to Viya 4. The data from the scan of your Viya 3.x environment will be included in the migration. Perform an inventory scan of the new Viya 4 environment. Perform a comparison scan to compare the inventory results from step 3 to those from step 1.   To run a Viya 4 to 3 comparison, set the COMPARISON environment variable to migration and then run the job. The scan will compare the most recent Viya 4 scan results to the scan results that were migrated from Viya 3.5. When the job is completed it is a good practice to reset the environment variable.   kubectl set env cronjob/sas-inventory-collector COMPARISON=migration kubectl create job --from=cronjob/sas-inventory-collector sas-inventory-comparison-job kubectl logs -l job-name=sas-inventory-comparison-job -f | klog kubectl set env cronjob/sas-inventory-collector COMPARISON-   Scan Results   SAS Viya 4 Inventory Report   The SAS Viya 4 Inventory Report surfaces the inventory results. The Status tab gives a summary of the scan results. Let's look at a couple of the useful Tabs on the report.   CAS Inventory   The CAS Inventory lists the defined CAS Libraries, the types of CAS Libraries, and the number of tables and file system size of each CAS Library.       Object Inventory   The Object Inventory tab lists all the objects in the SAS Viya Environment. You can filter on Internal Objects versus Customer Created Objects. The screenshot shows a count and list of the Visual Analytics Reports in the Viya environment.       NOTE: if you have a report named "Viya Inventory Report" your system was most likely migrated from Viya 3 and that is the Viya 3 version of the report.   SAS Viya Inventory Comparison Report   The SAS Viya Inventory Comparison report compares the two most recent scans or, if you have migrated from Viya 3.x it can compare to the scan results the Viya 3.x scan results. The report flags if the: ​ ​content is different.​ an object exists in the source deployment but not in the target deployment.​ an object exists in the target deployment but not in the source deployment.​ an object is different but only in the created or modified dates.     Wrap-Up   The inventory collector Cronjob makes it very easy to perform an inventory scan of a SAS Viya environment. The inventory reports surface the results of the scan. It is a recommended best practice to run a scan before and after any major changes in your Viya environment. For example, scan before and after migration from 3.5 or 4 to 4, an update of the Viya software, before a backup, and after a restore and before and after migrating content. The documentation for SAS Viya Inventory was recently updated and moved to the Tools area of the Viya Administration Guide.   Additional Information SAS Viya Inventory Documentation   Find more articles from SAS Global Enablement and Learning here. ... View more

Users of SAS Software are used to accessing external resources. Many of them store their SAS code and data in files located in directories in the file system. They then access the data in SAS libraries and path-based CAS libraries and the code from SAS clients. In SAS Education this is also the case for our workshop environments. In developing the Viya 4 Administration workshop our initial goal was to take our Viya 3.5 Administration environment and get functional equivalence in Viya on Kubernetes. In this post, I will look at how we secured our OS resources in Viya and Kubernetes.   In our Viya 3.x workshops we use a fictional company Gelcorp that has multiple business units Sales, HR, Finance, etc. Each BU has a group in LDAP and the Viya and OS resources are secured to the groups so the Sales cannot see Marketing and vice-versa.  In Viya 3.x:   Identities come from LDAP and users are grouped into Business Units (Sales, Marketing, etc.) System Security Services Daemon (SSSD) is configured on the LINUX server data and files are stored on a Network File System (NFS) I don't want to go into detail about how the security mode is implemented. To summarize the access pattern for a user is established based on their group and secondary group membership.   The permissions structure is shown in the table below. The impact of the settings is that a user can access resources for the groups they are members of, but cannot access resources for other groups. For example, Sally a member of the Sales group can access files in /gelcontent/gelcorp/sales/code but not in /gelcontent/gelcorp/hr/code.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Storage in Viya 4 Viya 4 runs on Kubernetes, it is different. The way you make the physical locations available is different. External storage can be made available using Persistent Volumes.  My colleague Steve Foerster also provides a really nice summary in this article.   In Viya 4 (like Viya 3.x) these OS resources must be secured so that they can be accessed by service accounts running SAS processes and by specific users and groups.​   Identities In Viya 4 identities can be either read from LDAP or loaded from a SCIM provider. In our case, we use the same LDAP provider as in our Viya 3 environment. The LDAP provider holds the users' POSIX attributes including primary and secondary group IDs. The data on our NFS share is secured using the identities of the provider.   In Viya 3.5 SSSD is the method we use to return identity information from LDAP to the OS. System Security Services Daemon is a system daemon. Its primary function is to provide access to local or remote identity and authentication. By default in Viya, SSSD is turned off for CAS and Compute, and all identity information is returned by the Identities service.   Let's look at one user on the system, Henrik to see how identity information is returned   Outside the pod on the NFS server, the id command shows the identity information of Henrik(4015) who has a Primary Group of sasuser(2003) and is a member of one secondary group HR(3001)         id Henrik           We can use a new pyviyatool getposixidentity.py to get the identifier information for Henrik that is stored in the Identities service. The names of the groups are not available from identities but the group IDs match what was returned by the id command.         /opt/pyviyatools/getposixidentity.py -u Henrik -o simplejson           What Identity Runs the Process? In order to determine how to secure the content on the filesystem we need to understand what identity runs the process. By default, it is different for CAS versus Compute.   Compute sessions:​ sessions are launched by the SAS Launcher Service and run as the requesting user by default.​ All Operating system access is by default by the user account.​ CAS sessions:​ by default, run as a process inside the CAS controller as the CAS service account (sas: uid=1001, gid=1001). ​ All Operating system access is by default by the service account.   To see this in operation and understand the implications for securing our OS content we can use the user Henrik and launch a compute and a CAS session. In Studio we run a SAS Program that will list our user ID, home directory, and the name of the compute pod that was launched. The program also starts a CAS Session.     To see the user running the compute session use kubectl to exec into a container in the pod that was launched to perform the SAS processing. Inside the container run the id command. The result is that the process is running as Henrik (4015) and the primary and secondary groups have been returned by the identities service.          kubectl exec -it $(kubectl get pod -l launcher.sas.com/requested-by-client=sas.SASStudio,launcher.sas.com/username=Henrik --output=jsonpath={.items..metadata.name}) -- bash -c "id"           For CAS we don't get a separate POD per session, we get a different process inside the CAS container on the CAS controller. We exec into the CAS container in the CAS controller pod. and execute the ps command we can see a CAS session running under the shared account (1001).   kubectl exec -it $(kubectl get pod -l casoperator.sas.com/node-type=controller --output=jsonpath={.items..metadata.name}) -c cas -- bash -c "ps -ef | grep 1002"   As you can see CAS and Compute behave differently. CAS, by default, uses the shared account, and Compute uses the requesting user's account.   Standardize Configuration For the compute server our setup is done. The OS permissions on the mounted directories will be honored because the compute sessions run as the requesting user and the Viya environment is using identity information from the same LDAP server as the NFS server.  For CAS we have some choices to make.   Firstly, we could ensure that in LDAP the Shared account has the group membership of all the business units via its secondary groups. If that were the case then the shared account would be able to access the data for the requesting user.  If the current shared account does not have this access pattern CAS can be configured to run as a different account.   Perhaps a more straightforward approach is to configure CAS so that the sessions run as the identity of the requesting user. To configure all CAS sessions to run as the requesting user's identity, set the CASALLHOSTACCOUNTS environment variable. Alternatively, adding specific users to the CASHostAccountRequired custom group will make CAS sessions for those users run as the identity of the requesting user. To test the difference let's use the last approach for the user Henrik.   Firstly, create the CASHostAccountRequired group and add Henrik as a member.   sas-viya --output text identities create-group --id CASHostAccountRequired --name "CASHostAccountRequired" --description "Run CAS as users account" sas-viya --output text identities add-member --group-id CASHostAccountRequired --user-member-id Henrik   Now run the SAS Program again and use the same command to check the process owner of the CAS session. Now we will see there is a process for Henrik(4015) in the CAS container of the CAS controller. The CAS session is running under the identity of the requesting user and the access pattern will be governed by that user's identity information.   kubectl exec -it $(kubectl get pod -l casoperator.sas.com/node-type=controller --output=jsonpath={.items..metadata.name}) -c cas -- bash -c "ps -ef | grep 4001"     With CAS and Compute standardized and the Viya deployment using the same LDAP server as the NFS server we can use the same group and secondary group membership techniques to secure resources that we do in a non Kubernetes Viya deployment.   This post has talked about securing data on NFS using POSIX attributes. In the cloud, there are many more storage options available and other options for identities where POSIX attributes are not available. That is a topic for another day.   For more information: Stuart Rogers blog on SAS Viya Secondary Groups and Posix Considerations CASHOSTACCOUNTREQUIRED and CASCLOUDNATIVE documentation   I would like to thank my colleagues @GillesChrzaszcz and  @StuartRogers, who were companions on this journey during the early releases of Viya 4.   Find more articles from SAS Global Enablement and Learning here. ... View more

POSIX attributes like UID and GID, and Secondary GID are vital elements of an operating system-level security model. In SAS Viya POSIX attributes can often be returned from the identity provider.  But what if your identity provider does not return POSIX attributes? In this post, I will look at the current default Viya behavior and how you can still use POSIX attributes to secure access to shared resources even when your identity provider does not return them.   The two main POSIX attributes we will focus on are UID and GID. UID and GID are the numerical equivalents of the username and primary group name referenced by the operating system and applications when determining access privileges. For example, every file and directory on a Linux system has an owning user and group.  When you want to share folders or files between groups of users or with another system you often use UID, GID, and Secondary GID information to determine who can access file system resources like SAS programs, CSV files, and SAS datasets.  In a previous blog post, I discussed securing OS resources that reside on the file system in SAS Viya 4.   Viya and POSIX attributes default behavior During the lifetime of SAS Viya, there have been some changes in how POSIX attributes are returned and managed. Let's first look at the current (2023.04) defaults. The identities service is the service that manages POSIX attributes. The following properties of the identities service determine the behavior:   sas.identities.identifier.disableGids : disables the lookup and generation of GIDs for users and groups. The default value is false sas.identities.identifier.generateGids: generates GIDs for users and groups. The default value is false. sas.identities.identifier.generateUids:  generates UIDs for users. The default value is true.   The default values for a Viya Deployment can be viewed in Environment Manager or surfaced using the configuration plug-in of the sas-viya cli.     sas-viya configuration configurations show --id $(sas-viya configuration configurations list --definition-name sas.identities | jq -r '.items[0]["id"]') | grep identifier         identifier.generateUids : true identifier.generateGids : false identifier.disableGids : false     As you can see, by default, with SAS Viya platform 2022.10 and later releases the identities service: always provides a generated UID allows you, as a SAS Administrator, to provide GID values   In our SAS Viya Administration course environment, we have an LDAP server that can return POSIX attributes and we keep the default settings for the identities.identifier properties. Let's use the user Delilah to demonstrate the resulting behavior. In SAS Environment Manager we see that Delilah is a member of the GELCorp Power Users, HR, and SAS Users groups.     We can check the POSIX attributes of Users and Groups using the identities plug-in of the sas-viya CLI. Here using the show-group command and --show-advanced option we see that HR has a GID of 3001, SAS Users has a GID of 2003 and Power Users has a GID of 3007. These values were read from our LDAP server because it can return the POSIX attribute and we are using the default setting identifier.generateGids : false.     sas-viya --output text identities show-group --id HR --show-advanced sas-viya --output text identities show-group --id sasusers --show-advanced sas-viya --output text identities show-group --id powerusers --show-advanced       Using the show-user command and the --show-advanced option we can see that the UID of the user Delilah is a generated value.     sas-viya --output text identities show-user --id Delilah --show-advanced       The default behavior when the UID is generated for a user is their primary GID (returned as just gid from the identifier endpoint) is always set to the same value as the UID. At first glance, this can be confusing. However, notice that secondaryGid values are populated with the GID’s read from LDAP for the groups that the user is a member of sasusers[2003], powerusers[3007], and HR[3001]   The default settings where GID is provided but UID is generated are useful when accessing data where a file system security model already exists and is developed based on Secondary GIDs. There might be times when the actual UID is important to the security mode. In these cases, we can just update the value of identifier.generateUids to true. In our case where LDAP does provide the POSIX attributes the UID values will then also be read from LDAP.   Generating POSIX Attributes In Viya environments where the identities service is configured to use SCIM, and in many where it is configured to use Active Directory, POSIX attributes are not returned. In this situation, where the identity provider cannot return the POSIX attributes, the SAS Viya Identities service will, for each user, automatically generate a UID and optionally a GID.   To generate the POSIX attributes the identities service will:​​   use the identity type (either USER or GROUP) along with the identity name  (e.g Delilah) and generate an SHA hash Map the SHA hash onto numerical space to generate the UID and GID values   Let’s look at the attributes for Delilah in another environment where the POSIX attributes are not available from LDAP. In this Viya on Azure environment, POSIX attributes are not available and Delilah has a slightly different group membership.     In this environment, in order to generate the POSIX attributes(UID and GID )the identities.identifier properties are set as follows. identifier.generateUids : true identifier.generateGids : true identifier.disableGids : false Let's look at what is returned for Delilah and the two Groups. The output shows we are returned both generated UID and GID values.     sas-viya --output text identities show-user --id Delilah --show-advanced sas-viya --output text identities show-group --id HR--show-advanced sas-viya --output text identities show-group --id PowerUsers --show-advanced       The generated UID and GID values can be problematic in managing access to file system content. If your filesystem content is shared with other applications and secured using existing UID, GID, and Secondary GIDs then the generated values will not match the existing values. As a result, many SAS processes will not be able to read the filesystem content. To address this problem Viya allows an administrator to provide POSIX attributes.   Providing POSIX Attributes There are two ways to use existing POSIX attributes. We have seen the first one where the attributes are read from the Identity Provider. Now we will look at the second approach where we load POSIX Attributes. POSIX attributes can be loaded to SAS Viya using the sas-viya CLI. This can be done for individual users, or you can bulk-load multiple users and groups from a CSV file.   The first thing we need to do if we want to switch to loading UID and GID is update the identities.identifier properties so that the POSIX attributes are not generated. In SAS Environment Manager under Configurations, select Definitions and sas.identities. Toggle identifier.generateGids and identifier.generateUids to the OFF position.     You can also make the same change using the configurations plugin of the sas-viya CLI.      If we look at Delilah after we change the properties notice, we no longer have POSIX attributes returned. At this point, our identity provider does not return POSIX attributes and we have not loaded any to the identities service (NOTE: the display is a little misleading the value for the UID and GID are not actually the number zero).     The identities plugin of the SAS Viya CLI can update individual UID/GID information. The sas-viya identities update-user command will update UID and primary GID  for a user, the update-group command updates the GID  for a group. For example, to update Ahmed:     sas-viya --output text identities update-user --id Ahmed --uid 4005 --gid 2003     This is useful for updating a single user or group. Using the CLI after 2022.10 you can use the bulkload-user-identifiers command to update multiple the UID and GID for multiple users from a CSV file. The format of the CSV file for users is:   Column 1: identity type (USER or GROUP) Column 2: identity name Column 3: numeric UID Column 4: numeric primary GID Here are a few lines from an example CSV.     USER,sasldap,1003,1003 USER,sasadm,2002,2002 USER,sastest1,2003,2003 USER,sastest2,2004,2003 USER,geladm,4000,2002 USER,Douglas,4001,2003 USER,Delilah,4002,2003         sas-viya --output text identities bulkload-user-identifiers -f "gelcorp_user_csv.csv"       Similarly, there is bulkload-group_identifiers command The format of the CSV file for groups is:   Column 1: identity type (USER or GROUP) Column 2: identity name Column 3: numeric GID Here are a few lines from an example CSV.     GROUP,HR,3001 GROUP,Sales,3003 GROUP,Finance,3004 GROUP,Managers,3002 GROUP,powerusers,3007     To bulk-load user POSIX attributes use bulk-load-group-identifiers     sas-viya --output text identities bulkload-group-identifiers -f "gelcorp_group_csv.csv"       NOTE: if you are using a release before 2022.10 this can also be accomplished with the setposixattributes.py tool of the pyviyatools.     If we run the show-user command now we will see that the UID and GID have been loaded from our CSV files to the identities service and the Secondary GIDs are available.     sas-viya --output text identities show-user --id Delilah --show-advanced       There is not an easy way to do this for all users using the sas-viya CLI but we can use one of the pyviyatools getposixidentity.py to get a list of the attributes for all users and groups.     /opt/pyviyatools/getposixidentity.py -o csv       Ongoing Maintenance When using this model of providing UID and GID information to identities service the administrator needs to be aware of the ongoing maintenance task of setting the UID and GID values when a new user is added to the identity provider. Failure to override the POSIX attributes when a new user is added will most likely result in the user not being able to access resources in the Viya system.   Wrap-up POSIX attributes like UID and GID, and Secondary GIDs are vital elements of an operating system-level security model. SAS Viya will often need to use POSIX attributes when accessing files on shared file systems. In this post, we looked at how Viya can access POSIX attributes. We looked at two scenarios, the first in which POSIX attributes were read from the LDAP identity provider. The second was where POSIX attributes were not available from the Identity Provider. In the latter case, we saw how Viya can generate POSIX attributes and how we can override the generated values. Thank you to my colleague Stuart Rogers for his input on this post. I have listed some additional resources below, in addition to these the Viya 4 Administration guide has an excellent section on "How to manage UIDs and GIDs with the identities service".   Additional Resources SAS Viya 2022.10 UID and GID Changes   SAS Viya Secondary Groups & POSIX Considerations   Accessing AWS S3 as NFS from CAS and SAS – Part 3   Group-based access to Azure Files in the new SAS Viya   ... View more