Let's go back to basics with some beginner-friendly authorization (permissions) administration in SAS Viya.   Out of the box, SAS Viya comes with a Public folder and a Public caslib to which all Authenticated Users have read-write access.   They are intended as fully public places where any user can share something with any other user. For many deployment scenarios, including classroom, lab and proof-of-concept deployments, this is great. But this simple openness is not always appropriate for larger enterprise or production deployments. A bank or government department whose SAS Viya deployment routinely deals with sensitive data and reports can't have places where users can share those things freely with anyone else in the organization - they would see such Public locations as insecure.   So, in this post we'll look at the simple steps you can take to reduce Authenticated Users ' access to both the Public folder and the Public caslib to read-only. And since I imagine this will be a pretty routine thing to do in many deployments, let's focus on a scripted way to do that, rather than using the SAS Environment Manager web interface. Helpful pyviyatools To make the scripts much shorter and easier to read, we'll use a couple of tools from our pyviyatools GitHub project. These are a collection of tools that the GEL team have written in python to help us with tasks for our workshop hands-on exercises, and to automate all sorts of useful things that were difficult or not currently possible to do with the SAS Viya 3 sas-admin and SAS Viya 2020.1 and later sas-viya CLIs when we wrote them. (One or two things have since become partially or completely redundant due to enhancements to the official SAS Viya CLIs, but the overwhelming majority still do something useful - to us at least - that you cannot easily do any other way).   You can read more about them in Gerry Nelson's post, Introducing the GEL pyviyatools. We have used or discussed pyviyatools in so many of our GEL posts since the release of SAS Viya, you may find something else interesting by searching the SAS Communities Library for pyviyatools. Default Public folder permissions for Authenticated Users Out of the box, when SAS Viya is first deployed, Authenticated Users are granted Read, Add and Remove permissions on the /SAS Content/Public folder itself, and Read, Update, Delete, Add and Remove permissions on the folder's contents:   In the SAS Environment Manager > Content page, right-click the Public folder to view or edit its permissions Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Default permissions for Authenticated Users on the SAS Content > Public folder. There are other permissions for SAS Administrators, not shown here.   This basically gives all users full read-write access to the Public folder and all of its contents, including everything in its subfolders. The only things not given to Authenticated Users are Update and Delete permissions to change or delete the Public folder itself, and Secure permission to change permissions on the Public folder or its contents. Make the Public folder read-only to authenticated users For sites that deal with sensitive content, you might prefer to reduce this to simply granting Authenticated Users Read on the folder, and Read on its contents, something like this:   Public Folder read-only to Authenticated Users   If you do, the Public folder will still exist and its contents will still be readable to everyone, but only SAS Administrators (whose permissions are not shown in the screenshot above) will have permission to put content into it. It's up to them whether they do or not. A SAS Administrator could of course delegate 'write' permissions to this folder to a trusted group.   For those readers who already understand all this, the following example script will change Authenticated Users' permissions on the Public folder from the former to the latter pattern shown above. It assumes several things: you have a current (Thursday 18 July 2021 or later) clone of the pyviyatools project in /path/to/pyviyatools - edit the pyviyadir variable to the path where your pyviyatools are you have a .authinfo file in your home directory with SAS Administrator credentials to authenticate to the sas-admin or sas-viya CLI. Here's the content of my .authinfo file, with the real password replaced by 'xxx': default user geladm password xxx you know the path to your sas-admin CLI (for SAS Viya 3) or sas-viya CLI (for SAS Viya 2020.1 and later) - these are actually somewhat interchangeable, but for simplicity's sake I'll use the predominant CLI name that came with each major version of SAS Viya; edit the sasviyacli variable to the path to your CLI Here it is:   #!/bin/sh pyviyadir=/path/to/pyviyatools $pyviyadir/loginviauthinfo.py #sasviyacli=/opt/sas/viya/home/bin/sas-admin # Example for Viya 3 users sasviyacli=/usr/bin/sas-viya # Example for Viya 4 users #Get "/Public" folder id, store in $id publicid=$($sasviyacli --output json folders show --path /Public | jq -r '.["id"]') echo "The /Public ID is" $publicid # Get the ID of the rule which grants permissions to Authenticated Users on the /Public folder objruleid=$($pyviyadir/getruleid.py -o json -u /folders/folders/$publicid/** -p "authenticatedusers" | jq -r '.items[].id') echo $objruleid # Get the ID of the rule which grants permissions to Authenticated Users on the /Public folder's contents contruleid=$($pyviyadir/getruleid.py -o json -c /folders/folders/$publicid -p "authenticatedusers" | jq -r '.items[].id') echo $contruleid # Show authenticated users’ current permissions on /Public folder itself before we make changes – should grant read, add and remove at this point $sasviyacli authorization show-rule --id $objruleid --details # Show authenticated users’ current permissions on /Public folder's contents before we make changes – should grant read, update, delete, add and remove at this point $sasviyacli authorization show-rule --id $contruleid --details # Delete both those rules $sasviyacli authorization remove-rule --id $objruleid $sasviyacli authorization remove-rule --id $contruleid # Create a single new rule to grant Authenticated Users Read on the /Public folder and on its contents $sasviyacli authorization create-rule --authenticated-users -o /folders/folders/$publicid/** --container-uri /folders/folders/$publicid --permissions Read -d "Authenticated users get read-only access to the Public folder and its contents" # Validate objruleid=$($pyviyadir/getruleid.py -o json -u /folders/folders/$publicid/** -p "authenticatedusers" | jq -r '.items[].id') contruleid=$($pyviyadir/getruleid.py -o json -c /folders/folders/$publicid -p "authenticatedusers" | jq -r '.items[].id') $sasviyacli authorization show-rule --id $objruleid --details $sasviyacli authorization show-rule --id $contruleid --details # Note that the rule id for both these tests is the same - they are both the same rule. We didn't really need to test it both ways, but it's reassuring that both tests give the same answer.   If you are familiar with this sort of script, feel free to skip the explanation which follows and move on to the next section which discusses the Public caslib. If not, read on!   To better understand the script above, let's recap how permissions work in the SAS General Authorization system. Endpoints and URIs Objects like folders and reports are identified by a REST endpoint identifying the type of object concatenated with and a globally unique ID number (a guid). Combined, these identify the object or resource uniquely, and are called a Uniform Resource Identifier, or URI. For example the Public folder in my current SAS Viya deployment is a folder (an object which is handled by the sas-folders service's folders REST endpoint: /folders/folders ), and has the object ID dba5473d-afb4-44d4-866a-9671ed5878c2 , and thus its objectUri is /folders/folders/dba5473d-afb4-44d4-866a-9671ed5878c2 . This guid and URI are basically meaningless in any other context, so I'm not revealing any great secret to show them here.   Most SAS Viya objects do not contain other objects, but a few such as folders do. These objects therefore have associated containers, can be referenced by their containerUri . I don't know if the containerUri has to always be the same as the objectUri , but I have not yet noticed a folder where these URIs differed. Permissions and Rules Permissions in the SAS Viya general authorization system are implemented behind the scenes with rules. A SAS Administrator can see the detail of these rules in the Rules page in SAS Environment Manager. Each rule can grant or prohibit (but not a mix of not both) any combination of seven permissions, to a single principle (typically all Authenticated Users , a group or a user), for a single SAS Viya object identified by its objectUri , or for that object's contents, identified by the object's containerUri . Or, in the special case when the permissions are the same for the object and its contents, the rule can target BOTH the object's objectUri and its containerUri at the same time. But a single rule cannot grant one set of permissions on an object itself, and a different set of permissions on the object's contents; to do that requires at least two separate rules (and sometimes, presumably for programmatic convenience, is implemented with more than two rules).   Authorization rules targeting the object's containerUri do not apply to the object itself, they only apply to its contents and to their contents too.   We saw in the screenshot above that the out-of-the box permissions granted to Authenticated Users on the Public folder and on its contents differ: Read, Add, Remove on the folder, Read, Update, Delete, Add and Remove on its contents. Does this pattern of permissions require one rule to implement, or two?   The SAS Viya general authorization system needs two rules to implement those permissions - one rule targeting the folder's Object URI to grant Authenticated Users Read Add and Remove, and a second rule targeting the folder's container URI to grant Authenticated Users Read, Update, Delete, Add and Remove permissions. How we may choose to secure the Public folder In order to secure the Public folder to make it read-only, let us delete both of these rules - we don't want Authenticated Users, or for the moment anyone else other than SAS Administrators, to have read-write access to the Public folder. We then intend to grant Authenticated Users Read access to the Public folder and its contents. Will that require one rule or two? Could we achieve this end result any other way?   The script above gives the answer to the first of those questions, near the end. As for the second question, yes, there are several other ways we could achieve this end result. Let's not worry about those for now, this way is perfectly sensible.   Our script above gets the object ID of the Public folder, and then uses a newly-updated pyviyatool called getruleid.py to get the IDs of general authorization rules targeting the folder's object URI and its container URI - we know now that there must be two separate rules.   It then deletes both rules, using $sasviyacli authorization remove-rule , and creates a single new rule, using $sasviyacli authorization create-rule , which grants Authenticated users the Read permission on both the Public folder, and its contents.   With the Public folder thus locked down, let's turn our attention to the Public caslib. Default Public CAS library permissions for Authenticated Users Out of the box, when SAS Viya is first deployed, Authenticated Users are granted 'Write' level permissions on the Public CAS library (caslib) which means all permissions except AlterCaslib and ManageAccess:   Default permissions for Authenticated Users on the Public caslib. Other permissions not shown here.   Much like the /SAS Content/Public folder, this basically gives all users full read-write access to the Public CAS library and all of its tables, unless different access controls are defined for any of its tables. The only things Authenticated Users are not allowed to do by default is change the name, path or options for the Public caslib, or alter its access controls. Make the Public caslib read-only to authenticated users For sites that deal with sensitive data, you might prefer to reduce this to simply granting Authenticated Users ReadInfo and Select on this caslib and its contents, something like this:   Authenticated Users access to the Public caslib reduced to the Read access level - ReadInfo and Select only.   If you do, the Public caslib will still exist and its tables and their data will still be readable to everyone, but only SAS Administrators (whose permissions are not shown in the screenshot above) will have permission to create, update and delete data in its tables. It's up to them whether they do or not. A SAS Administrator could of course delegate content such 'write' permissions to this caslib (and its tables) to a group of trusted users.   For those readers who already understand all this, here's a script that will change Authenticated Users' permissions on the Public caslib from the former to the latter pattern shown above, giving them only 'Read' access. It assumes: you have a clone of the pyviyatools project in /path/to/pyviyatools - edit the pyviyadir variable to the path where your pyviyatools are you have a .authinfo file in your home directory and you know the path to your sas-admin CLI and will edit the sasviyacli variable accordingly, as in the earlier code example. Here it is:   #!/bin/sh pyviyadir=/path/to/pyviyatools $pyviyadir/loginviauthinfo.py #sasviyacli=/opt/sas/viya/home/bin/sas-admin # Example for Viya 3 users sasviyacli=/usr/bin/sas-viya # Example for Viya 4 users caslib_name=Public json_file_cas_read=~/cas_read_permissions.json read_group_name=* # Authenticated Users is labelled * in CAS authorization # Show permissions for the group we're going to give Read level permissions to, before we change anything, formatted as text $sasviyacli --output text cas caslibs list-controls --server cas-shared-default --caslib ${caslib_name} | { head -1 ; grep "${read_group_name} (group)" ;} # Create a JSON file assigning CAS read-level permissions to the group in $read_group_name tee ${json_file_cas_read} > /dev/null << EOF { "items": [ { "identity": "${read_group_name}", "identityType": "group", "permission": "readInfo", "type": "grant", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "select", "type": "grant", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "limitedPromote", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "promote", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "createTable", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "dropTable", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "deleteSource", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "insert", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "update", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "delete", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "alterTable", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "alterCaslib", "type": "deny", "version": 1 }, { "identity": "${read_group_name}", "identityType": "group", "permission": "manageAccess", "type": "deny", "version": 1 } ] } EOF cat ${json_file_cas_read} # Grant the group read level permissions on their caslib_name, assuming the CAS super-user role (-su) $sasviyacli cas caslibs add-controls --server cas-shared-default --caslib ${caslib_name} --source-file ${json_file_cas_read} -su # Expected output: # The requested access controls in the source file "/home/cloud-user/cas_read_permissions.json" were applied to the caslib "Public". # Show permissions for the group we just gave Read level permissions to, formatted as text $sasviyacli --output text cas caslibs list-controls --server cas-shared-default --caslib ${caslib_name} | { head -1 ; grep "${read_group_name} (group)" ;}   To explain this script above, we need to know how permissions work in the SAS CAS Authorization system, which is used only for CAS libraries and tables (and CAS Action Sets, if we're being thorough).   The sas-viya CLI has a cas plug-in, which provides great command-line tools for viewing and changing CAS library and table access controls. After using the loginviyauthinfo.py pyviyatool, which uses the .authinfo file in your home directory to authenticate, the next thing this script does is list the existing access controls on the Public caslib, by running the sas-viya --output text cas caslibs list-controls command with arguments identifying the CAS server and library. It pipes the text-formatted output of this command through a head -1 command to show the header row of the list-controls output, and also through a grep command to filter the rest of the output to only show rows for * (group) which is how Authenticated Users are shown in this output.   The sas-viya (or sas-admin ) CLI's cas caslibs subcommand has various options for exporting and importing complete or partial sets of access controls as JSON files. The quickest way to update many access controls at once is by passing in a JSON file, so next we create one, with the 'Read' access level, where all permissions are denied to the chosen group, except for ReadInfo and Select which are granted. This JSON file is called ~/cas_read_permissions.json in the example code above, but you can call it whatever and place it wherever you like - it does not have to go in the current user's home directory ( ~/ ). /tmp is another reasonable choice since we use it immediately after creating it and are unlikely to re-use it later without modification.   The example script then runs sas-viya cas caslibs add-controls , to apply the access controls specified in that JSON file to the table, replacing any existing access controls for the same principal ( * =Authenticated Users).   Finally, the script lists and formats the access controls for our group ( * =Authenticated Users), to show how they are now set.   The end result should be that your Public CAS library has only Read-level permissions for Authenticated Users, as shown in the screenshot earlier in this section. Why not just delete them? This seems a reasonable question.   You certainly could just delete the /Public folder and Public caslib, and see what happens: does something in your deployment break because it depends on one of them, or does something re-create them if they are absent (e.g. when a service starts up)? In my experience and limited testing, deleting them caused no issues, but my personal testing definitely did not cover all use cases one could reasonably anticipate! You should not rely on my casual observation for a business-critical deployment.   As 'out of the box' locations, it's possible that some customer-written application or solution they use expects those locations to exist, or may need to use them for a legitimate purpose, for sharing non-sensitive data under the careful eye of a responsible data or content administrator in future. It is also possible that some migration process in future will expect one or both to be present, and fail if they are not.   In any case, I don't think removing the /Public folder and Public caslib completely is really necessary. Making them read-only to Authenticated Users greatly reduces the likelihood they will be used irresponsibly. There is little potential for harm if you put nothing in them, so I would just do that and otherwise leave them be.   See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more

A feature in the SAS Launcher which limits the number of concurrent compute sessions for each user was added in SAS Viya 2021.1. The idea is to provide denial-of-service protection by preventing users from maliciously creating a large number of compute sessions. But it also provides a feature that SAS Administrators have long asked for, to help them manage fair use of compute resources.   In this post, we'll see how it works in practice, how to tell what your current limits are, and how to change the limits or remove them completely.     SAS Launcher's User Process Limits  Let's suppose a user called Henrik has already started all the compute servers he is allowed to, by opening multiple SAS Studio sessions in separate browser tabs. Each of them started as normal, and each has a SAS Compute server process running under the SAS Studio compute context:   A SAS session has been started successfully in the SAS Studio compute context.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Then Henrik, who is not being very considerate of other system users, opens SAS Studio in yet another browser tab, and waits while the button top right displays a wait symbol indicating that it is attempting to start another compute server for him:     Waiting for a SAS Compute Server session to start.   After a much longer wait than for the previous successful sessions, SAS Studio displays a red 'x' in a circle, indicating that the compute server failed to start. If Henrik hovers his pointer over the button, a tooltip gives only a minimal hint about what is wrong: "An attempt to launch a SAS session has failed":   Failed to launch a SAS session in the SAS Studio compute context.   Unless Henrik has access to the SAS Studio web application logs (see below), this is all the information Henrik gets about why his latest compute session failed to start, when many others previously started normally. He may learn than he can start only so many sessions at a time, but it would be helpful for the SAS Administrator to let users know there is a limit, and what that limit is.   If Henrik then closes one of his existing compute sessions, so that the compute server pod which hosts that session terminates, he will find that he is able to open a new compute session again.     See the real error message In whatever logging solution is available to the administrator (here we are using Kibana from the SAS Viya Monitoring for Kubernetes project), the administrator can search for log messages from logsource: sas-studio-app and level: ERROR , to find a more informative error message explaining why the compute session did not start for Henrik:   We can find a more informative error message in Kibana     The full error message reads:   com.sas.studio.exceptions.ServerOperationException: Unable to create compute server session. Unable to complete the launch request, max number of active processes has been reached for the user: user=Henrik limit=2   So while the user session limit is n, the reason for a failure to launch a user's SAS session n+1 may not be immediately obvious to that user in SAS Studio. But it is something the administrator can explain.     See the current limits on SAS Compute Sessions As a Kubernetes administrator, or at least a user with access to the Kubernetes cluster and permission to describe the SAS Launcher deployment in the SAS Viya namespace, you can see the current user process limit by running this kubectl command:   kubectl describe deploy sas-launcher | grep SAS_LAUNCHER_USER_PROCESS_LIMIT   Expected output in an environment where the default values have not been changed:   SAS_LAUNCHER_USER_PROCESS_LIMIT: 10 SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED: true   By default users are allowed to have 10, 25 or 50 simultaneous SAS Compute processes running at a time, each in their own separate Kubernetes pod. The limit depends on the deployment size per the table below.   There is also a new custom group called LauncherSuperUsers , empty by default. It allows the administrator to define a separate limit for users who e.g. need more sessions. For users who are directly or indirectly members of this custom group, the SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT limit applies instead. To see all four values, run:   kubectl describe deploy sas-launcher | grep SAS_LAUNCHER.*USER_PROCESS_LIMIT   Expected output:   SAS_LAUNCHER_USER_PROCESS_LIMIT: 10 SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED: true SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT: 15 SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT_ENABLED: true   The USER and SUPER_USER limits can be disabled by setting SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED and/or SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT_ENABLED to false. The default size of the two limits depends on the size of your deployment - we can infer from this that my SAS Viya deployment is a 'Small' deployment:   User limits for compute sessions by deployment size   How to change SAS Launcher User Process Limits We have seen that the SAS Launcher user process limits are set by four environment variables set for the Kubernetes deployment for sas-launcher. Here is how the SAS Viya Administration documentation explains them:     Variable Description SAS_LAUNCHER_USER_PROCESS_LIMIT Default: 10 (Small), 25 (Medium), 50 (Large) Modifies the process limit by users who do not belong to the LauncherSuperUsers custom group. SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED Default: TRUE Disables or enables the process limit for users who do not belong the LauncherSuperUsers custom group. If the value is FALSE (disabled), then the user can have an unlimited number of active processes running. SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT Default: 15 (Small), 35 (Medium), 65 (Large) Modifies the process limit for users who belong to the LauncherSuperUsers custom group. SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT_ENABLED Default: TRUE Disables or enables the process limit for users who belong to the LauncherSuperUsers custom group. If the value is FALSE (disabled), then the user can have an unlimited number of active processes running.   So, how can we change them?   Quick temporary method You can temporarily change the values using kubectl. The example below sets SAS_LAUNCHER_USER_PROCESS_LIMIT to 2, but you can set any of the other environment variables the same way:    kubectl set env deploy sas-launcher SAS_LAUNCHER_USER_PROCESS_LIMIT=2   Note that this command will update the value in the deployment now, but it won't change the SAS Viya deployment manifest - usually called site.yaml - that was used and may well be used again to deploy SAS Viya into the cluster. Any changes to Kubernetes environment variables you apply like this would likely be lost next time site.yaml is applied to the cluster. Longer but permanent method To set these SAS Launcher environment variables more permanently, add a patchTransformer to your kustomization.yaml file, re-run kustomize to re-generate your site.yaml file, and re-apply it to your Kubernetes cluster. We'll create a separate patchTransformer yaml file, and then reference it from our kustomization.yaml, in the same way as many other Kubernetes-level configuration changes are made to SAS Viya deployments.   If your SAS Viya deployment directory is at /home/cloud-user/project/deploy/ and the deployment is called gelcorp , you can create the patchTransformer yaml file something like this:   tee /home/cloud-user/project/deploy/gelcorp/site-config/launcher-user-process-limit.yaml > /dev/null << EOF ############################################################################### # Kustomize patch configuration to change the process limit per user for # sas-launcher. # See README.md for more information. ############################################################################### apiVersion: builtin kind: PatchTransformer metadata: name: sas-launcher-processes-per-user-limitation patch: |- - op: replace path: /spec/template/spec/containers/0/env/1 value: name: SAS_LAUNCHER_USER_PROCESS_LIMIT value: "2" # value: "25" # value: "50" target: version: v1 group: apps kind: Deployment name: sas-launcher EOF   That done, reference it in the kustomization.yaml file by adding a line like this, if it is not already present, in the transformers section:   - site-config/launcher-user-process-limit.yaml   Here's a bit of script that may do that for you, using grep and yq:   _FILE=~/project/deploy/gelcorp/kustomization.yaml _MATCH_LINE="site-config/launcher-user-process-limit.yaml" if grep -qe "$_MATCH_LINE" $_FILE ; then echo "NOTE: Found \""${_MATCH_LINE}"\" in \""${_FILE}"\". We should not insert it again." ; else echo "NOTE: Did not find \""${_MATCH_LINE}"\" in \""${_FILE}"\". Adding it." ; printf " - command: update path: transformers[+] value: ${_MATCH_LINE} " | yq -I 4 w -i -s - $_FILE fi   Here's part of my kustomization.yaml file containing the inserted line:   . . . configurations: - sas-bases/overlays/required/kustomizeconfig.yaml transformers: - sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml # TLS . . . - site-config/launcher-user-process-limit.yaml patches: . . .   Then, when you are ready, rebuild site.yaml and apply it to your cluster in the usual way, with commands something like:   kustomize build -o site.yaml then kubectl apply -f site.yaml   When you have done that, you should see that the value of SAS_LAUNCHER_USER_PROCESS_LIMIT has changed to whatever value you set it to. Do demonstrate the effect for this blog post, I set it to the unusually low value of 2, so that's what I see when I re-run:   kubectl describe deploy sas-launcher | grep SAS_LAUNCHER_USER_PROCESS_LIMIT   Expected results after applying the change by either of the above two methods:   SAS_LAUNCHER_USER_PROCESS_LIMIT: 2 SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED: true   Restore the original value If you changed the values using the quick temporary method, using kubectl set env , you can either run another similar command to restore the variables you changed to their previous values, or you could re-apply your site.yaml file to do so.   If you changed the values by adding a reference to a patchTransformer to your kustomize.yaml file, restore the default value by removing or commenting the line referencing site-config/launcher-user-process-limit.yaml from your kustomization.yaml and repeating the steps above to re-create and reapply site.yaml .     Reference   See the documentation for this feature in the SAS Viya Administration guide > Servers and Services > Programming Run-Time Environment > Programming Run-Time Servers > SAS Launcher Service > Configure Denial-of-Service Protection for detail: a https://go.documentation.sas.com/doc/en/sasadmincdc/v_014/calsrvpgm/n00002viyaprgmsrvs00000admin.htm#p0dvcvne6of1run1b1sl0z1j303r   See you next time! ... View more

System logs reveal what happens behind the scenes in a SAS Viya deployment. But SAS Viya system logs include sensitive data, so administrators need to manage who can access those logs.   When K8s clusters have more than one Viya deployment, administrators may need a way to manage access to logs by deployment, so that a user of one deployment can view logs from that deployment, but not from other deployments.   Three scripts included in the SAS Viya Monitoring for Kubernetes project, and documented in the SAS Viya Administration Guide, create and delete Kibana backend roles, role mappings and users, so that the new users are restricted to seeing logs from a single SAS Viya namespace. These scripts greatly simplify what may otherwise be an unfamiliar and somewhat confusing procedure. In this article we'll explore where to find those scripts, how to run them, and we'll see what they do.   Where to find the logging module administration scripts   If you have not already cloned the SAS Viya Monitoring for Kubernetes project from GitHub, here's how:   git clone https://github.com/sassoftware/viya4-monitoring-kubernetes.git   Cloning this project creates a new subdirectory called viya4-monitoring-kubernetes in the current directory. There is a main README.md file in that subdirectory, and there are further README.md files in the monitoring and logging directories below it. Follow the instructions in the SAS Viya: Administration documentation, or those in the README.md files to deploy and configure the two modules of this project, for Monitoring and Logging. Administration scripts for the logging module of SAS Viya Monitoring for Kubernetes live under the logging/bin subdirectories of that directory:   [cloud-user@pdcesx11147 ~]$ ls -1 viya4-monitoring-kubernetes/logging/bin/ change_internal_password.sh . . [omitting several other scripts] . security_create_rbac.sh security_delete_rbac.sh user.sh Of these, the last three are the ones we will focus on in this article.     Two Kubernetes Namespaces: a viya_namespace and a logging_namespace All of the scripts or commands covered in this article require the name of a Kubernetes namespace as a parameter, or in an environment variable, or both. The topic covered by this article involves two sorts of Kubernetes namespace, so to avoid confusion:   I will refer to a namespace containing a SAS Viya deployment with the placeholder name viya_namespace. For example, my Kubernetes cluster has two SAS Viya deployments, named gelcorp and gelinstitute . So, I would substitute one or other of those namespace names for viya_namespace wherever a command refers to it. I will refer to a namespace containing the SAS Viya Monitoring for Kubernetes project's logging components as the logging_namespace. In this same cluster, the logging components of SAS Viya Monitoring for Kubernetes are deployed in a namespace called ops4viyalog . So, I would substitute ops4viyalog for logging_namespace. You will need to make similar substitutions in your environment if you try this.   Read the documentation and README.md files carefully. They often just use the placeholder 'namespace' without specifying which one they are talking about, because it is nearly always the viya_namespace. The viya_namespace and logging_namespace placeholders are not likely the actual strings you would use in your commands, unless that is what your namespaces are actually called!     How to use the Kibana security Role Based Access Control (RBAC) scripts  As described in the SAS Viya Administration documentation for Logging and Monitoring, under 'Set Up Access for Kibana Users', these scripts are intended to be used as follows, from within the logging/bin subdirectory.   Script and arguments Use security_create_rbac.sh viya_namespace Set up Kibana role-based access for a SAS Viya namespace called viya_namespace. This script creates the  search_index_viya_namespace  role and the  viya_namespace_kibana_users back-end role. security_delete_rbac.sh viya_namespace Delete the roles and mappings created by security_create_rbac.sh, for a SAS Viya namespace called viya_namespace. user.sh create --namespace viya_namespace [--user username] [--password password] Create a Kibana user with the viya_namespace_kibana_users backend role, granting it access to Kibana, but only to see logs from the namespace called viya_namespace. The default value for  --user username  is the  namespace name (i.e. whatever you put in place of viya_namespace). The default value for --password password is the  username . user.sh delete --user username Delete a Kibana user that was created by this user.sh script.   Important: As the documentation explains, these scripts do not validate the viya_namespace value . If no such namespace exists, the scripts will still run quite happily, and set up roles, role mappings and users in Kibana that refer to the nonexistant namespace.   Tip 1: Notice that the security_create_rbac.sh and security_delete_rbac.sh scripts currently take the namespace as an argument, without --namespace  before it. The user.sh script has more parameters, and so the namespace parameter must be preceded by --namespace  (or -ns ). Tip 2: Run any of the commands with a -h or --help  argument to see more detailed usage information. Run user.sh create --help or user.sh delete --help for usage information specific to the user create and delete options. Tip 3: You should run the scripts as a user who has administrator kubectl access to your logging_namespace (or at least, as a user who has access to get the Kibana admin username and password secrets from your logging_namespace). All the scripts use kubectl commands to get the Kibana administrator username and password, and to set up Kubernetes port-forwarding. Tip 4: Several of the scripts output a message like this:   INFO You may see a message below about a process being killed; it is expected and can be ignored. The scripts create a temporary Kubernetes port-forwarding process, to enable them to communicate with the Kibana services in your K8s cluster. When the scripts are done, they kill the port-forwarding process. This is normal.   The LOG_NS environment variable Most, or all scripts in the logging/bin directory expect an environment variable called LOG_NS to be set. It should contain the logging_namespace name mentioned earlier. I have used logging_namespace as a placeholder for this name in the examples below - replace it with the actual name of your logging namespace! Here are three ways to set this environment variable: Option 1: export LOG_NS=logging_namespace [command] Option 2: export LOG_NS=logging_namespace ; [command] Option 3: Define a USER_DIR environment variable, referring to a directory containing one or more specific user configuration files for the SAS Viya Monitoring for Kubernetes project.   See https://github.com/sassoftware/viya4-monitoring-kubernetes/blob/master/logging/README.md#customize-the-deployment for a detailed description of the configuration files for the logging component of this project, and how they can be used. (The Monitoring component of this project, outside the scope of this article, can also be customized - see https://github.com/sassoftware/viya4-monitoring-kubernetes/blob/master/monitoring/README.md#customize-the-deployment.   Note: You may already have something like this set up, from when you deployed the monitoring and logging modules. This option is perhaps a bit overcomplicated if you only wish to set one environment variable for the project. But is a good choice when you need to customize many of them.   Examples Here is an example for a SAS Viya deployment in a viya_namespace called gelcorp . The SAS Viya Monitoring for Kubernetes logging module has already been installed in the cluster, in a logging_namespace called ops4viyalog . (ops4viya was a pre-release name for what is now the SAS Viya Monitoring for Kubernetes project). The whole GitHub project has been cloned into the current user's home directory as described earlier.     Set up Kibana role-based access for the gelcorp viya_namespace   cd ~/viya4-monitoring-kubernetes/logging/bin export LOG_NS=ops4viyalog ./security_create_rbac.sh gelcorp Expected output:   INFO Helm client version: 3.2.1 INFO Kubernetes client version: v1.18.6 INFO Kubernetes server version: v1.18.6 INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env Creating access controls for namespace [gelcorp] [Sun Jan 10 10:24:14 EST 2021] INFO Security role [search_index_gelcorp] created [201] INFO Security rolemapping created between [search_index_gelcorp] and backend-role(s) [gelcorp_kibana_users]. [201] INFO Security rolemapping created between [kibana_user] and backend-role(s) [gelcorp_kibana_users]. [200] INFO You may see a message below about a process being killed; it is expected and can be ignored. ./security_create_rbac.sh: line 221: 15807 Killed kubectl -n $LOG_NS port-forward --address localhost svc/v4m-es-client-service :$ES_PORT > $tmpfile Access controls created [Sun Jan 10 10:24:30 EST 2021] ============================================================================================================================ == Assign users the back-end role of [gelcorp_kibana_users] to grant access to log messages for [gelcorp] namespace and Kibana == ============================================================================================================================   Set up a Kibana user named 'gelcorp_admin' for the gelcorp viya_namespace   cd ~/viya4-monitoring-kubernetes/logging/bin export LOG_NS=ops4viyalog ./user.sh create --namespace gelcorp --username gelcorp_admin --password password-goes-here Expected output:   INFO Helm client version: 3.2.1 INFO Kubernetes client version: v1.18.6 INFO Kubernetes server version: v1.18.6 INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env INFO Attempting to create user [gelcorp_admin] and grant them access to namespace [gelcorp] [Sun Jan 10 10:31:33 EST 2021] INFO User [gelcorp_admin] created. [201] User [gelcorp_admin] added to internal user database [Sun Jan 10 10:31:34 EST 2021] INFO You may see a message below about a process being killed; it is expected and can be ignored.   You can run user.sh create ... more than once to create more users.   Delete a Kibana user named 'gelcorp_admin' There is no need to specify the viya_namespace or password when deleting a user.   cd ~/viya4-monitoring-kubernetes/logging/bin export LOG_NS=ops4viyalog ./user.sh delete --username gelcorp_admin   Expected output:   INFO Helm client version: 3.2.1 INFO Kubernetes client version: v1.18.6 INFO Kubernetes server version: v1.18.6 INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env INFO Attempting to remove user [gelcorp_admin] from the internal user database [Sun Jan 10 10:38:42 EST 2021] INFO User [gelcorp_admin] deleted. [200] User [gelcorp_admin] removed from internal user database [Sun Jan 10 10:38:43 EST 2021] INFO You may see a message below about a process being killed; it is expected and can be ignored.   It's recommended to delete all users you created for the viya_namespace before deleting the Kibana role-based access for that Viya namespace. If you do not, there will be one or more users left behind who have roles assigned to them that no longer exist. This may leave a Kibana user who appears to be set up properly, yet is unable to access any log data.     Delete Kibana role-based access controls for the gelcorp viya_namespace   cd ~/viya4-monitoring-kubernetes/logging/bin export LOG_NS=ops4viyalog ./security_delete_rbac.sh gelcorp   Expected output:   INFO Helm client version: 3.2.1 INFO Kubernetes client version: v1.18.6 INFO Kubernetes server version: v1.18.6 INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env Deleting access controls for namespace [gelcorp] [Sun Jan 10 11:02:02 EST 2021] INFO Security rolemappings for [search_index_gelcorp] deleted. [200] INFO Security role [search_index_gelcorp] deleted. [200] INFO Security rolemapping deleted between [kibana_user] and backend-roles [gelcorp_kibana_users, gelcorp_kibana_ro_users]. [200] INFO Security rolemapping deleted between [kibana_read_only] and backend-roles [gelcorp_kibana_users, gelcorp_kibana_ro_users]. [200] INFO Security rolemapping deleted between [cluster_ro_perms] and backend-roles [gelcorp_kibana_users, gelcorp_kibana_ro_users]. [200] INFO You may see a message below about a process being killed; it is expected and can be ignored. ./security_delete_rbac.sh: line 173: 25351 Killed kubectl -n $LOG_NS port-forward --address localhost svc/v4m-es-client-service :$ES_PORT > $tmpfile Access controls deleted [Sun Jan 10 11:02:21 EST 2021] =============================================================================================================== == You should delete any users whose only purpose was to access log messages from the [gelcorp] namespace == ===============================================================================================================   When you delete the Kibana role-based access controls for a viya_namespace, there is no validation check to see if any users are still granted those role-based access controls.     Results: Kibana RBAC for one SAS Viya namespace In another environment, I have two deployments of SAS Viya, in namespaces called gelcorp and gelinstitute .   The overall Kubernetes Admin user ('admin') can see log messages for both these Viya deployments, and from other K8s namespaces too:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I used the scripts to set up the RBAC for gelcorp , and created a user called gelcorp_admin, as per the examples above. The gelcorp_admin can only see log messages for the gelcorp namespace:     Extended results: Kibana RBAC for two SAS Viya namespaces   In the same environment, I then used the same scripts to set up Kibana roles and role mappings for my second Viya deployment, in the namespace gelinstitute :   cd ~/viya4-monitoring-kubernetes/logging/bin export LOG_NS=ops4viyalog ./security_create_rbac.sh gelinstitute   I also created a second Kibana user called gelinstitute_admin to use the access controls for gelinstitute :   cd ~/viya4-monitoring-kubernetes/logging/bin export LOG_NS=ops4viyalog ./user.sh create --namespace gelinstitute --username gelinstitute_admin --password password-goes-here   As expected, this user can only see log messages for the gelinstitute namespace:     Finally, I created a third new Kibana user called gelboth_admin, initially with access to gelcorp logs. You can create as many users as you wish with access to one set of Viya logs in this way. But I then ran the create script a second time to try to grant it access to gelinstitute logs.   cd ~/viya4-monitoring-kubernetes/logging/bin export LOG_NS=ops4viyalog ./user.sh create --namespace gelcorp --username gelboth_admin --password password-goes-here ./user.sh create --namespace gelinstitute --username gelboth_admin --password password-goes-here   The second time, the user.sh script exits with an error, because the user already exists:   INFO Helm client version: 3.2.1 INFO Kubernetes client version: v1.18.6 INFO Kubernetes server version: v1.18.6 INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env INFO Attempting to create user [gelboth_admin] and grant them access to namespace [gelinstitute] [Sun Jan 10 11:33:13 EST 2021] RROR There was an issue creating the user [gelboth_admin]; user already exists. [200] This makes perfect sense. But it is easy to finish setting up our gelboth_admin user, to give it access to the gelinstitute logs as well. Log into Kibana as the admin, and navigate to the Security page. It has an icon which looks like this:     Under Authentication Backends, click Internal Users Database. [caption id="attachment_42396" align="aligncenter" width="300"]   Kibana Security page, with Internal User Database button highlighted   In the Internal Users Database page, we see all three of our gel*_admin users (gelboth_admin, gelcorp_admin and gelinstitute_admin):       Notice that gelboth_admin currently has only one backend role, gelcorp_kibana_users. We want to also grant this user the gelinstitute_kibana_users backend role. So, click the Edit button for gelboth_admin. The under Backend Roles, click the 'Add backend role' button.     You can enter the name of the backend role to add by hand: gelinstitute_kibana_users. Or you can copy-and-paste it from the output of the security_create_rbac.sh script, e.g. from near the end of the line of output like this (you may have to scroll right to see it):   INFO Security rolemapping created between [search_index_gelinstitute] and backend-role(s) [gelinstitute_kibana_users]. [201]   Click the Submit button at the bottom of the page. Now, in the Internal Users Database page, our user gelboth_admin has both of our new Backend Roles:     As a result, gelboth_admin can now see log messages in Kibana for both Viya deployments, gelcorp and gelinstitute , but not for any other Kubernetes namespaces:     This works exactly as expected. I hope this will help you manage which users get access to log messages from each of your SAS Viya deployments on Kubernetes. See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more

Hosting SAS Viya in the cloud will soon become more common than hosting it on bare OS, if it is not already. Some cloud providers and their partners offer tools and services to help you implement disaster recovery (DR) capabilities.   Plus, the fact that SAS Viya 2021.1 and later runs on Kubernetes introduces new considerations for DR beyond those we had for SAS Viya 3.x on bare OS. So while SAS Viya 2021.1 and later on Kubernetes is not significantly more or less vulnerable to disaster than any previous release of SAS, these two things merit an update on considerations for DR in SAS Viya.   The slide below outlines considerations that are specific to cloud hosting and to Kubernetes:       SAS Viya on Kubernetes has specific technical requirements for Disaster Recovery. Your DR site's Kubernetes environment must have the same SAS Viya products at the same version of SAS Viya as your production site. The documentation says you must also use the same namespace name in the DR cluster, but we think this is more of a preference than a requirement, since it is quite straightforward to restore a Viya 4 backup to a different namespace name, and namespace names and DNS aliases are easily decoupled - while it may be common practice to have your Viya deployment's namespace name (production, test, gelcorp etc.) as a part of its DNS alias, it is certainly not a requirement to.   It must also be deployed on the same version of Kubernetes in the same cloud infrastructure, so failing over from one Azure AKS cluster to another Azure AKS cluster is okay, but we do not support failing over from an Azure AKS cluster to an Amazon EKS cluster as part of a DR process.   Disaster Recovery for SAS Viya on Azure Kubernetes Services (AKS)         Microsoft Azure Site Recovery's best practices for DR in AKS recommend using their DNS-based load balancer Azure Traffic Manager to route traffic to either your primary or secondary AKS cluster, hosted in different Azure regions. It can interconnect two clusters to enable communication between them for data replication. It also has features for replicating container images between the Azure Container Registry in each Azure region where you have an AKS cluster, and the best practice guide above discusses several aspects of replicating storage. But it remains the technical architect's responsibility to figure out how to replicate SAS Viya state data between clusters in different regions in a way that will satisfy the customer's desired RPO and RTO. More on this below.   Start-up Arpio (see below) is also developing a capability for Azure, not yet released.     Disaster Recovery for SAS Viya on Amazon Elastic Kubernetes Services (EKS)   Arpio       Arpio is a start-up based in Durham, NC (near Cary) who offer security-conscious replication of a production AWS environment in one region to a second 'recovery' region. SAS Cloud is moving towards using Arpio for DR capability, but at this point it's fairly new to me. See arpio.io/how-it-works/ for their marketing.     AWS CloudEndure       Amazon AWS CloudEndure is their DR offering for EKS-hosted SAS Viya deployments. Documentation for using CloudEndure for EKS clusters seems to be light, but the general principles for what needs to be replicated and how traffic needs to be directed to whichever region is currently hosting your services is the same as in Azure.   The unique selling point for CloudEndure seems to be cost reduction, in that it uses low cost staging machines on the DR site when the production site is healthy. In the event of a disaster in Production it scales up the machines on the DR site to full size, ready for business use.   However, CloudEndure may not always meet the security requirements for some of our more data security-conscious customers.     Google Cloud Platform and OpenShift We have not so far identified similar services specifically developed to support Disaster Recovery for Google Cloud Platform or OpenShift. If you know of any, please tell me, as we'd like to cover them in our content.     Further reading on SAS Disaster Recovery In addition to Disaster Recovery practices outlined in the SAS Viya documentation, there is a SAS Disaster Recovery Policy for SAS Viya 3.4. There is not yet an SAS Disaster Recovery Policy specifically for SAS Viya on Kubernetes. Rob Collum's SAS Communities post on Contemplating disaster recovery for SAS from 19 July 2018 is also well worth reading.   Further considerations for DR   General The resources listed above aim to explain: That disaster recovery is not a feature of SAS software, it is a capability you must design, create and maintain How to define a disaster in practical, measurable, objective terms which could potentially be instrumented for automated decision making That resilience to disasters is based on eliminating single points of failure, that hosting on a single site creates such a single point of failure, therefore you need two sites BUT network latency means deployments that span sites will not work, so Prod/DR must be active/passive and synchronised They also discuss how important it is that you agree with your business stakeholders: What the risks and potential costs of a disaster impacting SAS services are to the business If and how you will design your services to mitigate those risks so far as is practically possible e.g. through eliminating single points of failure for both storage and compute, load balancing and horizontal scaling for high availability as well as for performance What you would do in the event of a disaster, in order to provide business continuity, and What you would do following the disaster, to achieve full recovery to normal business operations How you will keep a production site and a DR site in synchronization How long a period of work and data loss can be tolerated How quickly you must recover from a disaster, to ensure business continuity and what implication this has for automating the failover process to reduce delays How often you will test your DR processes, and how much prior warning you and the business will get of a test. Should you consider something as extreme as Netflix's Chaos Monkey? Hosting SAS Viya on Kubernetes is significantly more complex than hosting SAS Viya on bare OS. Backing up and replicating to another environment is correspondingly more complex too. Data Survey the data in YOUR SAS Viya environment which needs to be synchronised or replicated (e.g. by being periodically backed up and copied). These item are present in all SAS Viya deployments: SAS Viya deployment directory (containing kustomize.yaml and site.yaml) and overlays SAS Infrastructure Data Server (pg_dump) SAS Configuration Server content CAS permstore (it should not be necessary to back up CAS configuration files separately in Viya 2020.1 and later, since they are re-created at each CAS server startup) Optional but recommended: log and metrics data to assist investigations into the circumstances of a disaster Here are some examples of data which may be present in some deployments: Kubernetes Persistent Volumes (PVs) used for mounting filesystem-based user data (e.g. NFS shares, user home directories), PVs for block storage, source data etc. If SAS ingests data from upstream data sources which cannot be obtained again, consider whether you should synchronise or replicate that to DR, or whether you need to keep upstream data available for a time in case the SAS Viya system has an outage and loses some data it pulled. If the upstream systems can re-supply data, it may be unnecessary. Administration and operation scripts for the cluster Mirror Docker Image repos and other deployment assets for all deployed software to allow (re-)deployment of the same release of software in both PROD and DR Git repos used for e.g. CI/CD/DevOps tasks Customizations to third-party logging, monitoring and alerting tools such as saved searches, dashboards, custom metrics, custom alerts, if not already stored in e.g. a git repo that is backed up and/or replicated Any other customer or project data/scripts/assets stored with or in the production environment that aren't covered by the categories above Each separate data type will usually require its own element of your overall approach to data synchronisation from production to DR, and you likely need to sync them all. Plan for the capability to synchronize data in both directions: following a disaster and failover to your DR site, that site temporarily hosts real business activity. When your main production site is back up and available again, you must be able to synchronize all these types of data from your DR site back to your main production site, and perform what is sometimes called a 'fail back' but better described as a switch of services back from DR to production.     Please share your experience of DR for SAS Viya with the community I have found fewer resources, case studies and examples of partially or wholly successful implementations of disaster recovery capabilities than I hoped to find, given how important we know DR is for many of our customers.   If you have designed, implemented or operated a DR capability for a SAS Viya deployment I would very much like to hear from you. There is no substitute for real-world experience, and we want to hear about yours, good or bad. Please leave a comment below, or email me at David.Stern@sas.com to share your stories, designs or case studies on this topic. Thank you in anticipation.     My thanks to Peter Muirhead, Gerry Nelson, Scott McCauley and Rob Collum for their review of earlier drafts of this post, and for their helpful feedback. Any errors or shortcomings are mine. ... View more

The way the new SAS Viya (2020.1 and later) log messages are collected and made available to SAS administrators is very different from the way they were handled in SAS Viya 3 or SAS 9. In this post, I'll describe the overall idea of how logging works in SAS Viya 4, without getting into too much technical detail.   SAS Viya services run inside containers, which are in turn deployed in pods in a Kubernetes cluster. You could draw it like this:    A Viya service runs inside a Docker container, which is in a Kubernetes pod, on a node, in a Kubernetes cluster.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Let's look at how those log messages get from the process to the administrator's attention.   Logs go to stdout and stderr, not SAS .log files   Generally speaking, processes running inside containers in Kubernetes pods cannot write log files or anything else to a persistent file system. It is a feature of Kubernetes that most pods are supposed to be able to be automatically started and stopped as needed, with as many copies of each pod as Kubernetes is told to run, that can be run on the node which Kubernetes thinks they can run best. The services provided by those pods should continue to operate, with users, and to some extent administrators, being mostly unaware of all this orchestration. Just writing log files to a fixed location on the host machine's disk does not make much sense.   A few pods like those serving CAS, SAS Infrastructure Data Server (Postgres), SAS Configuration Server (Consul) and SAS Message Broker (Rabbitmq) are allocated Kubernetes 'persistent volumes' which are basically managed filesystem storage, so that they can save their state when the pods are stopped. But those pods are in a minority, and all SAS services write log messages. Using persistent volumes to write out log files is definitely not the right way to do things in Kubernetes.   Instead, SAS Viya 4 deployments follow the Kubernetes standard pattern: services and processes write log messages out to stdout and stderr. No more named log files in a specific path. That may take quite some getting used to!   Sidenote: This post is concerned with what happens to log messages once they have been written out by a service or process. SAS Administrators also need to think about managing log thresholds which control the least severe classification of log message that gets written out by SAS Viya services. If we lower the threshold, e.g. to debug or trace, the service will write out more - sometimes many more - log messages under the same workload. If we increase the threshold e.g. to warning or error, the service will write out fewer messages but we miss some useful details. This aspect of logging is something a Kubernetes administrator will know far less about than a SAS administrator should.   Kubernetes picks those messages up, and saves them in pod logs. Pod logs can be accessed from the Kubernetes command line interface kubectl, or via REST calls to the same Kubernetes API that kubectl and more or less everything else in Kubernetes uses. But out of the box Kubernetes doesn't do much more with them than that. What's more, SAS administrators might not have access to kubectl or the REST API for the cluster where Viya is hosted.   Even the Kubernetes administrator, who presumably knows the kubectl command line interface very well, will not much like using kubectl to search through hundreds of thousands of log messages every day. So, we need something else to give SAS Administrators and other users a way to get to those log messages.   There are many ways to manage and surface Kubernetes logs. One is to combine three tools as something generically called the EFK stack. There are many off-the-shelf and self-assembly ways to combine these tools, but as a general approach it is popular and well-established.     The EFK Stack: Elasticsearch + Fluentd/Fluent Bit + Kibana   An EFK stack collects log messages from the pod logs on each Kubernetes node. Some implementations of EFK stacks use a component called Fluentd; other implementations, including ours, opt to use its lightweight sibling, Fluent Bit. Fluent Bit in our EFK stack performs a similar function to sas-watch-log in the SAS Viya 3 operations infrastructure.   In our implementation, Fluent Bit runs on every Kubernetes node, and streams pod logs over the Kubernetes internal network into Elasticsearch, a text-indexing-and-searching engine (dare I say database?) which stores the log messages and their contextual data. This contextual data includes fields such as timestamp, source process, container, pod, namespace, node, severity, process id, process owner etc. Elasticsearch stores all this for each log message, along with highly-searchable indexes. The last part of the 'stack', Kibana, reads data from Elasticsearch, applies management policies and permissions to it, and presents the collated, re-formatted and better-organised log messages to administrators a nice web interface.   Customers who have a lot of experience with Kubernetes, or have already incorporated Kubernetes into their IT ecosystem before they deploy SAS Viya 4 will almost certainly already have some sort of logging solution for their cluster. It may be a flavor of the EFK stack. Those customers will likely use their existing logging toolset to manage and work with SAS Viya logs. But they may need our help to fine-tune the way SAS log messages are collected, re-formatted, stored in separate indexes per SAS deployment, and presented in Kibana or another tool's reports.   Customers who do not already have Kubernetes and are adopting it at the same time as deploying SAS Viya 4 may choose to use a lightly-customised deployment of the EFK stack, which constitutes the logging part of our SAS Viya Monitoring for Kubernetes project.(https://github.com/sassoftware/viya4-monitoring-kubernetes)   Logs are created by every SAS Viya service, in each container in each pod, on every node where a SAS Viya pod runs in the cluster. They are gathered first in Kubernetes pod logs.    Every process, including every Viya process, in every container in every pod in the Kubernetes cluster writes out log messages to stdout and stderr. These are collected first by Kubernetes in pod logs.   When our SAS Viya Monitoring for Kubernetes project's logging solution is deployed to the cluster, it puts a Fluent Bit component on each node in the cluster. It uses a Kubernetes object called a daemonset to ensure each node gets one instance of Fluent Bit.   The EFK stack is deployed on the Kubernetes cluster, placing Fluent Bit on each node. Fluent Bit streams log messages from the pod logs into Elasticsearch, from where they can be surfaced in Kibana.   The EFK stack is deployed on the Kubernetes cluster, placing Fluent Bit on each node. Fluent Bit streams log messages from the pod logs into Elasticsearch, from where they can be surfaced in Kibana.[/caption]   As part of the SAS Viya Monitoring for Kubernetes project, our version of the EFK stack has had some significant configuration work done on the Fluent Bit component, to adjust the format and metadata for some log messages and quite a bit else besides. We have also configured Elasticsearch's ingest pipeline to store log messages from different SAS Viya deployments in different Elasticsearch indexes - its equivalent of tables.    Log messaged watched by Fluent Bit are streamed to an Elasticsearch ingest pipeline, which splits into separate indexes per SAS deployment   Because log messages from different SAS Viya deployments in the same Kubernetes cluster are put in different Elasticsearch indexes, and because Kibana and Elasticsearch allow different users and group to be granted different levels of access to indexes, you can control who sees logs for each SAS Viya deployment in the cluster. Where two customers called e.g. ACME and ABCO are deployed in the same Kubernetes cluster, you can manage administrator access to them in the way illustrated in this diagram:      The ACME and ABCO SAS administrators can be granted access to see log messages for their own deployments, and nothing else in the Kubernetes cluster. However, a Kubernetes cluster administrator may be granted access to see and manage log messages from both SAS Viya deployments, as well as other sets of logs from other pods in the cluster.       I really like that the logging subsystem provided by the EFK stack in SAS Viya Monitoring for Kubernetes does not depend on any SAS service to continue working. In SAS Viya 3, the Operations Infrastructure depended on CAS, among other things, to surface log messages in SAS Environment Manager. This means that if CAS is down, you should see error messages in Kibana from CAS, whereas in Viya 3, you likely would not see error (or any!) messages in the Logs page of SAS Environment manager.   Our SAS Viya Monitoring for Kubernetes implementation of the EFK stack does not require an additional license fee for SAS customers who choose to use it. However, the components do of course consume cluster resources and are therefore not entirely 'free' to run - those cluster resources have a cost. Cloud vendors generally make it quite easy to see what the running costs are for each namespace. This allows the customer to see the running costs of the logging solution they deploy and use.   Find more articles from SAS Global Enablement and Learning here.   See you next time! ... View more

"Can you send me the logs?"   A familiar phrase to anyone who has ever asked for help with an error or performance issue in SAS.   For SAS 9 and earlier, and for SAS Viya 3.x, this usually means finding and sending one or more '.log' files on the filesystem.   However, in the latest releases of SAS Viya - sometimes referred to SAS Viya 4 to distinguish it from SAS Viya 3.x - the process of getting logs to send to someone else is a little different. SAS Viya services and jobs running on Kubernetes do not write out log files in the way they did in earlier releases. Instead, services and jobs stream log messages to stdout and stderr, and it's up to Kubernetes and the Kubernetes administrator to decide what - if anything - to do with them.   This is definitely a better design pattern for a scaleable, dynamic, resilient application running on Kubernetes. But we have to learn how to work with logs in a new way.   Just to clear up a potential point of confusion, in this post I'm not talking about the SAS logs you get when you run a SAS program that might contain data step and macro code, procedures and so on. The way those logs work in SAS Viya 4 has evolved slightly, but when you run a program in SAS Studio or SAS Model Studio, you still get the logs and the rest of the results much as you did before. SAS batch jobs, and even interactive 'line mode' in SAS Viya 4 still return SAS logs much as earlier versions, even if the details have changed a little. In this post, I'm talking about the logs output by server processes, running in Kubernetes pods, that your client application (like SAS Studio, SAS Visual Analytics, SAS Environment Manager or the SAS Command Line Interface) connects to. Five tools for exporting logs There are several way to export SAS Viya log messages. Each of them have some pre-requisites in terms of software that needs to be installed, or credentials you need to use them. And none of them is the 'best' way by itself, so it's worth knowing them all: Mirantis Lens The SAS Viya Log Download Tool from the SAS Viya 4 Administration Resource Kit The SQL Workbench in Kibana from SAS Viya Monitoring for Kubernetes kubectl logs The experimental Export Logs tool ( getlogs.sh ) in SAS Viya Monitoring for Kubernetes Mirantis Lens If you are a Kubernetes administrator, or at least have some access to the Kubernetes cluster, you can export container logs from Mirantis Lens, a very slick and capable user interface for Kubernetes clusters. This is definitely the easiest way, but it has two limitations: You can only export logs for one container at a time. You can only export logs from the ~10 MB cache that Kubernetes stores for each running or recently ended pod. If the pod containing the service whose logs you want to see is very busy and outputs a lot of logs, old messages can be dropped from Kubernetes' cache before you get to see them, to make way for more recent log messages. You only get the pod logs, not the log documents that SAS Viya for Kubernetes streams to Elasticsearch, enriched with additional fields for better context. At least you know (or can easily find) that context by looking at the pod's attributes (name, node etc). when you download its log messages. To use Mirantis Lens, you need a Windows, MacOS or Linux machine, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.   This clip shows how:   How to export logs in Lens. Look for the download button, bottom right, in the log view. Select the full-screen icon to see a larger version. The SAS Viya Log Download Tool from the SAS Viya 4 Administration Resource Kit The SAS Viya 4 Administration Resource Kit is an excellent collection of tools for administrators and anyone who deploys or maintains SAS Viya 4. One of them, the SAS Viya Log Download Tool is a utility that will download all (or a selection) of pod logs from a single Kubernetes namespace. You should specify a SAS Viya namespace, and you can optionally specify: which pods to download logs from (otherwise all pods) a limit to the number of messages to download from each pod  (otherwise all available messages) where to save out the logs (otherwise, it writes them to a directory named  sas-k8s-logs in the current working directory) Its limitations are that: As with Mirantis Lens, you can only export logs from the ~10 MB cache that Kubernetes stores for each running or recently ended pod. Being a command tool, not everyone will find SAS Viya Log Download Tool quite so easy to use as a visual UI like Mirantis Lens. You only get the pod logs, not the log documents that SAS Viya for Kubernetes streams to Elasticsearch, enriched with additional fields for better context. At least you know (or can easily find) that context by looking at the pod's attributes (name, node etc). when you download its log messages. However, it's strengths are that: people who are comfortable with a command line, and have a beginner's familiarity with python and kubectl will find it easy to use, and you can script it, with all the usual benefits this brings: the potential to reduce manual effort through automation, aid repeatability/reduce mistakes and the fact that a script can be scheduled, triggered in response to an event, or run ad-hoc. To use SAS Viya Log Download Tool you need a bash shell environment with python 3, the kubectl Kubernetes CLI, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.   Here's an example of how you could use it - there are lots more in the Log Download Tool's documentation:   python3 viya-ark.py download-pod-logs -n sas --output-dir="/path/to/report/" The SQL Workbench in Kibana from SAS Viya Monitoring for Kubernetes Our SAS Viya Monitoring for Kubernetes project provides an excellent monitoring and logging solution for SAS Viya on Kubernetes. The logging module is SAS' take on the EFK stack, using Elasticsearch, Fluent Bit and Kibana, and is customized for SAS Viya. It is easy for a Kubernetes administrator to install into the cluster, and it's great for collecting, categorising, exploring and displaying SAS Viya log messages.   This logging solution's limitations are that: It is a containerised Kubernetes application. The Kubernetes administrator has to deploy it into the cluster before you can use it. The Elasticsearch database can get quite large. By default Elasticsearch keeps logs for 3 days - this may be longer than Kubernetes keeps some of the pod logs, but shorter than we became used to in earlier versions of SAS, where logs may have been kept permanently until the administrator did something to clear them down, or for e.g. 30 days. To export log messages you have to write queries by hand and submit them through Kibana's SQL Workbench interface. This requires knowledge of both SQL and the way SAS Viya logs are organized in Elasticsearch indexes. The SQL Workbench user interface is occasionally a bit flaky, especially when exporting larger numbers of log messages. However, for exporting logs, this tool has some very significant advantages of over the other two methods: Our logging stack, specifically our custom-configured Fluent Bit agent adds additional context fields to the raw log message 'documents' output by services to pod logs. This context includes the namespace, the pod, the container name and other things that are essential to understanding the message properly. It also 'fixes' some quirks in the way services output or categorize lot messages. Those issues are not fixed in the pod logs. It also has Kubernetes event messages, and sends them to Elasticsearch as if they were log messages, with logsource=KUBE_EVENT. This is very useful for debugging pods that won't start - and whose services therefore don't output any log messages. Elasticsearch doesn't keep the last 10 MB of messages from each pod. Instead, it keeps the last 3 days of messages from each pod. It is sometimes the only place you can still see older messages from pods which output a lot of messages, and cycle through their 10 MB pod log cache quickly. Conversely, for pods which do not output many messages, sometimes this means there are fewer messages from that pod in Elasticsearch than in the pod logs. It is easy to retrieve log messages from more than one pod at once. In fact, this is the default, unless you specify a pod name (or something which has the same effect as specifying the pod name) in the SQL where clause. Since SAS Viya has many instances of services with multiple pods, this allows you to see what happened in any of those pods instead of having to look in each of them one at a time. Think about messages output by CAS worker pods on large CAS deployments - you might not want to look through the logs for each worker node individually. It also allows you to see a unified series of log messages spanning multiple services in the same set of results. SAS Viya is a highly distributed application and in some cases this is the only way to see logs for an interesting sequence of events that span several services. To use Kibana's SQL workbench, you only need a web browser and login credentials! Well, you need SQL skills and a bit of knowledge of our indexes too.   This screenshot shows an example SQL query, and the Download button:   After running a query in Kibana's SQL Workbench page, you can download the results as a CSV, or in the other formats shown. Select the image to see a larger version. Mobile users: To view the image, select the "Full" version at the bottom of the page.   Three things to note: Most of Kibana takes time filters and shows results in your local browser's time zone. Behind the scenes, Elasticsearch works in UTC, and the SQL Workbench does too. You will need to convert between your local time zone and UTC if you filter on @timestamp in your SQL where clause. Currently, you are limited to querying one Elasticsearch index in SQL Workbench. We configure our logging solution to have one index per namespace, per UTC day. So queries spanning UTC days need to be split. By default, SQL queries retrieve 200 rows. Add a LIMIT statement to the very end of your SQL query to change this, e.g. to a higher number. kubectl logs You can of course export pod logs using kubectl logs , so it deserves a brief mention. kubectl logs is easily the least capable tool of all covered in this post. Its main advantage is being simple, and Kubernetes admins already have it and likely know how to use it.   Its limitations are that: Being a command tool, not everyone will find kubectl easy to use. You can only export logs from one pod, or pods in the same group It has limited filtering options. You only get the basic pod logs, the same as with Mirantis Lens and the SAS VIya Log Download Tool. No extra context fields, no KUBE_EVENT messages, and none of the other fixes we make to the log documents in SAS Viya Monitoring for Kubernetes. To use kubectl , you need it to be installed in your shell, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.   Search the web for more on how to use it, but for example, this command would output logs for the sas-logon-app to a file called sas-logon-app.ndjson:   kubectl logs -l app=sas-logon-app --namespace=sasviya > sas-logon-app.ndjson The experimental Export Logs tool ( getlogs.sh ) in SAS Viya Monitoring for Kubernetes The latest release of SAS Viya Monitoring for Kubernetes at the time of writing is release 1.0.5, published on 15th March 2021. It includes a new, experimental tool, called the Export Logs tool. I have not had much time to experiment with it yet, and will write more about it in a future post, but it looks really nice. See the documentation here. To shamelessly copy and paste from that documentation:   To download the log messages, run the  getlogs.sh  script. This is the syntax for the script: getlogs.sh -ns | -- namespace   namespace   [query options] [time period options] [output options] [connection options] [other options] The  -- namespace   namespace  option is required. This value specifies the Kubernetes namespace corresponding to the SAS Viya deployment for which you want to obtain log messages. It runs in Bash on Linux, but not currently on MacOS. Please bear in mind that this new tool is experimental, and is likely to change over the next few releases of the project.   Its limitations are that: Being a command tool, not everyone will find SAS Viya Log Download Tool quite so easy to use as a visual UI like Mirantis Lens. It behaves a little quirkily when you limit the number of rows you want, but your query will span multiple Elasticsearch indexes, and you may get more rows than you asked for as a result. Bonus Stern! As my wife keeps telling me, more Sterns are definitely a good thing. However, this particular Stern (no relation, honest!) can be very useful both in live analysis of logs, and capturing of relevant info: https://github.com/wercker/stern. From its readme: " Stern allows you to  tail  multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging." When you want to watch something live, and you know the specific pod or even the container that will generate the message (even if there are many copies of that pod running on your cluster), Stern is terrific for observing those log messages in real time.   As with kubectl logs, the SAS Viya Log Download Tool and Lens, you only get the basic pod logs. You won't see the extra context fields, KUBE_EVENT messages, or other improvements made to log JSON documents in SAS Viya Monitoring for Kubernetes. Conclusions So, which is best? Honestly, there is no tool among this set which is 'the best' in all use cases.   I really like the SAS Viya Monitoring for Kubernetes logging stack, and Kibana. They have the best log data, and the richest tools for filtering, but SQL Workbench is a somewhat advanced tool, with a bit of a learning curve.   Lens is just gorgeous to use, but it's log export is pretty basic and limited to one pod at a time. For simple stuff, it's perfect.   The SAS Viya Log Download Tool is great for dumping all logs to a zip file. If you don't know which log you need someone to look at, this is the tool for you.   The last three also only offer the raw, un-enhanced pod logs with less context, and you have to hope the messages you want are within the most recent 10 MB cache.   I'm excited to learn more about the new Export Logs tool in SAS Viya Monitoring for Kubernetes, and every Kubernetes admin knows kubectl logs already.   So really, they are all useful, and I encourage you to become familiar with all these methods, so that when someone asks "Can you send me the logs?", your answer should be "Yes!"   See you next time! ... View more

Last time we took a first look at Azure Monitor Logs for AKS clusters , and saw how to create an Azure log analytics workspace from the Azure CLI and then enable monitoring for your AKS cluster. Today I'll assume you've already done that, and also followed the next steps in that post to confirm log data is being collected in Azure Monitor Logs.   Having spent a little more time with these Azure-native logging tools, I'm fairly confident you're better off using SAS Viya Monitoring for Kubernetes. It's prettier, it has more features, it is much easier to use, and it works just as well in an Azure AKS cluster as anywhere else. But if you really want to, or really have to use Azure's own tools for viewing SAS Viya logs in tables or charts, or if you want see a bit about how much they cost, read on!   Still here? This time we're going to create some distinctive log messages, and explore these log messages in a bit more depth with Azure Monitor Logging. We'll use a couple of example Kusto queries which format the logs a bit better, and make them more readable. We'll view log data in a summary chart, and see how much of SAS's money I spent on Azure Monitoring logging writing a blog post. All the following code examples have a green bar at the side if they are commands you can run (perhaps after modifying them to suit your needs):   Green bars are for commands you can run   Finally, this post includes the code of an example Kusto SQL query from the Microsoft Azure Monitor Logs site, and that is shown in a box with a purple bar:   Purple bars are for Kusto queries   Create some distinctive log messages   Open SAS Studio in your cluster. The URL for SAS Studio in my cluster (username: sukdws, region: eastus) was http://sukdwsvk.eastus.cloudapp.azure.com/SASStudio. Obviously your URL for SAS Studio will differ from mine.   Log on to SAS Studio and if necessary skip the setup and the tour. When SAS Studio finishes opening, open a New SAS Program, and run this code to put some distinctive text in the SAS logs:   %put The quick brown fox jumped over the lazy dog;   It should look something like this:   SAS Studio after writing a distinctive log message   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Find the distinctive log message in Azure Monitor Logs   If you successfully followed the steps in my previous post, including the steps to 'Confirm log data is being collected in Azure Monitor Logs', then this distinctive log message output by the compute server (compsrv) will have been captured in the Azure Log Analytics Workspace. Let's go find it! Last time we ran a very basic Kusto query, like this:   // List container logs per namespace // View container logs from all the namespaces in the cluster. ContainerLog |join(KubePodInventory| where TimeGenerated > startofday(ago(1h)))//KubePodInventory Contains namespace information on ContainerID |where TimeGenerated > startofday(ago(1h)) | project TimeGenerated ,Namespace , LogEntrySource , LogEntry   This time, add one extra line to it, as the second-last line:   // List container logs per namespace // View container logs from all the namespaces in the cluster. ContainerLog |join(KubePodInventory| where TimeGenerated > startofday(ago(1h)))//KubePodInventory Contains namespace information on ContainerID |where TimeGenerated > startofday(ago(1h)) | where LogEntry contains "The quick brown fox" | project TimeGenerated ,Namespace , LogEntrySource , LogEntry   Later on, I'm going to refer to the query above as the 'original, basic query'.   The number of results you get may vary a bit, but the time I tried this while writing this blog post, I got 45 (!) rows of results, all containing that string. The longer I waited after running the program in SAS Studio before I ran or re-ran this query in Azure Monitor Logs, the more rows of results I seemed to get:   Azure Monitor Logs Results of basic query to search for “The quick brown fox”     As before, these are clearly logs from our deployment and the message we're interested in is there, but it is still not very easy to see:   which container or service generated the log message the severity of the message (DEBUG, INFO, WARNING, ERROR etc.) where there is one the text of the log message (in some cases) there are definitely duplicate rows Let's replace the original, basic query in the New Query tab in Azure Monitor Logs with the following new, improved query:   let startTimestamp = ago(1h); KubePodInventory | where TimeGenerated > startTimestamp | project ContainerID, PodName=Name, Namespace | distinct ContainerID, PodName, Namespace | where Namespace == "lab" | where PodName contains "sas-launcher" | join(ContainerLog | where TimeGenerated > startTimestamp) on ContainerID | where LogEntry contains "The quick brown fox" // The next line parses the JSON doc in LogEntry and stores the result in an dynamic variable called 'log' // Doing this allowing us to extract individual fields from that object | extend log=parse_json(LogEntry) | extend Level=log.level | extend Source=log.source | extend Message=log.message | project TimeGenerated, Namespace, Source, Level, Message, PodName, LogEntry, LogEntrySource | order by TimeGenerated desc   Let's run this new, improved Kusto query first, see the results it produces. Afterwards, we will break it down and compare it with the original basic query to understand how each works. When you run this new, improved query, the results should look something like this:   Azure Monitor Logs improved Kusto query   This is better than the original, basic query because: There are only two rows of results, not dozens. We fixed the duplicates issue and filtered the results better. The log source, level (i.e. SAS's severity indicator) and message are much easier to see. They have their own columns in the results table. We can even see which pod generated the messages - they are from a pod named sas-launcher-long-guid where long-guid is some long id number. In fact, the Kusto query is filtered to only show messages from pods whose names contain the string "sas-launcher". This helped eliminate some unwanted rows, apparently from a search.appendIndex call in CAS. When querying Azure Monitor Logs, Kusto queries can be written in either Kusto query language (or KQL, which is the 'preferred' language) as T-SQL select statements (for compatibility with tools that can't easily be converted to use KQL).     Elements of our improved log query   Let's break down the things in this query that improve on the previous one.   Original, basic query New, improved query Notes   let startTimestamp = ago(1h); 1. The new improved query defines a variable containing the start time, of 1 hour ago. This offers a slightly more convenient way to change the time range over which log messages are displayed, and doing so once, when the time is used in two places in the new query. But other than that convenience, it is not materially different to setting the time inline, like the original query does. ContainerLog KubePodInventory | where TimeGenerated > startTimestamp | project ContainerID, PodName=Name, Namespace | distinct ContainerID, PodName, Namespace | where Namespace == "lab" | where PodName contains "sas-launcher" 2. The Original query takes both the ContainerLog table and the KubePodInventory table, and joins them before any filters are applied. This works, but it requires matching of more rows from both tables, and could thus potentially be slightly slower, though in practice I didn't notice a meaningful difference in query time. The new query filters down the rows in BOTH tables before they are joined, keeping only rows from either table generated since since the start time stamp, and also filtering the KubePodInventory rows to keep only those for the namespace and pod we are interested in, and taking a distinct set of the resulting container, pod and namespace(s) to avoid duplicate results. This makes for a much more efficient, deduped query. |join(KubePodInventory| where TimeGenerated > startofday(ago(1h))) //KubePodInventory Contains namespace information on ContainerID | join(ContainerLog | where TimeGenerated > startTimestamp) on ContainerID 3. Both queries the join the (filtered or unfiltered) rows from KubePodInventory and ContainerLog on ContainerID. We discussed the time filters in these lines already. |where TimeGenerated > startofday(ago(1h))   4. The original query then further filters the results to keep only those from the most recent day. The new query doesn't need to do that, it is done already. | where LogEntry contains "The quick brown fox" | where LogEntry contains "The quick brown fox" 5. Both queries filter for messages containing "The quick brown fox" in the same way.   // The next line parses the JSON doc in LogEntry and stores the result in an dynamic variable called 'log' // Doing this allowing us to extract individual fields from that object | extend log=parse_json(LogEntry) | extend Level=log.level | extend Source=log.source | extend Message=log.message 6. The new query then uses a function called parse_json to extract the values out of the JSON document stored in the LogEntry field. It temporarily stores these in a dynamic object we chose to name 'log', before extracting the values we are interested in, from log.level, log.source and log.message. Try experimenting with extracting other values from this field if you like! | project TimeGenerated ,Namespace , LogEntrySource , LogEntry | project TimeGenerated, Namespace, Source, Level, Message, PodName, LogEntry, LogEntrySource 7. The project operator in Kusto selects the columns in the results so far to include, rename or drop, and can also be used to insert new computed columns. Here, both queries use it to select which columns to keep.   | order by TimeGenerated desc 8. The new query orders the resulting log message rows, newest first. Remove the 'desc' if you want oldest first.   Search the web for other examples of functions and operators you can use in Kusto to modify log queries to display log messages in a tabular format that is most useful to you.     Summary Charts in Azure Monitor Logs   Our resident logging observability guru Greg Smith shows how you can create rudimentary charts using the Kusto summarize operator and the Chart display tab of Azure Monitor logs, in the AZURE_LOG_ANALYTICS_WORKSPACES.md readme file in the SAS Viya Monitoring for Kubernetes project. It's worth a read.   To demonstrate, we'll modify our new, improved query slightly, commenting out (//) the line that filters to just show log messages to only have those where PodName contains "sas-launcher", and adding a new summarize line at the end. This 'chart query' is then:   let startTimestamp = ago(1h); KubePodInventory | where TimeGenerated > startTimestamp | project ContainerID, PodName=Name, Namespace | distinct ContainerID, PodName, Namespace | where Namespace == "test" //| where PodName contains "sas-launcher" | join(ContainerLog | where TimeGenerated > startTimestamp) on ContainerID // The next line parses the JSON doc in LogEntry and stores the result in an dynamic variable called 'log' // Doing this allowing us to extract individual fields from that object | extend log=parse_json(LogEntry) | extend Level=log.level | extend Source=log.source | extend Message=log.message | project TimeGenerated, Namespace, Source, Level, Message, PodName, LogEntry, LogEntrySource | order by TimeGenerated desc | summarize msgcount=count() by tostring(Level), tostring(Source)   The results table doesn't look very impressive:   Azure Monitor Logs chart query table     But if you click the Chart tab, next to results, just above the table, you get a simple histogram showing the same data which is quite nice:   Azure Monitor Logs Chart     To change what is shown in the chart, you have to manually edit your Kusto query. For example, I would be tempted to do things like: only keep rows where Level is 'error' or 'warning', or exclude rows where source is 'cas'   But really there's no one right thing to do next with a chart like this. It depends what you're interested in - use the tool to follow a train of thought and see if it takes you anywhere useful or interesting.   The charting capabilities we've found so far in Azure Monitor Logs are however nowhere near as good as those offered by Kibana, as used in the SAS Viya Monitoring for Kubernetes project. For this, and many other reasons, that project offers a far more capable logging solution for SAS Viya running on an Azure AKS cluster than Azure Monitor Logs does, and we would recommend you use it if you can.     If you don't need it, disable monitoring in Azure to save money   Collecting log data in Azure Monitor Logs can run up quite a bill if you aren't paying attention. The chart below shows the Azure Cost Management tool, and within it a cost analysis of all my resource groups and the log analytics service for the roughly five-hour period that I had a collection up and running and monitoring enabled, on a practically idle SAS Viya deployment:   Azure Log Analytics costs for about 5 hours   That means I spent a little under USD $8 for five hours of log analytics service, to write this blog post. This Azure Monitor service is charged by GB of data ingested, and per GB of data retained. The larger cost by far of those two is for data ingested. For a whole day at the pay-as-you-go rate, we might extrapolate this to guess that the cost might be around $30-40, for a nearly-idle Viya deployment like the one I used. If the deployment was actually being used for work, I can only assume the cost would go up very significantly from that. The pricing page does explain how you can use 'Capacity Reservations' to reduce these costs, but even so, this is something you will want to keep an eye on and budget for.   If you only enabled the monitoring add-ons in Azure to try this out, but do not plan on using them any further, please save some money by disabling the addons, and (optionally) deleting the Log Analytics Workspace too. My previous post has a section at the end titled 'Cleanup: Disable Monitoring and Remove the Log Analytics Workspace' which shows you how to do this.    See you next time! ... View more

SAS has released an open-source project in GitHub called SAS Viya Monitoring for Kubernetes, to provide observability tools for monitoring, logging and alerting in SAS Viya 4. I would encourage customers who get SAS Viya 4 to also deploy SAS VIya Monitoring for Kubernetes, because it's really good. However, it is not the only option for monitoring, logging and alerting.   There are several other open-source and commercial logging solutions that work with Kubernetes (k8s) deployments. And all of the big three cloud hosting providers have their own native monitoring, logging and alerting solutions, which also support k8s.   Today we'll take a first look at one of these, which will be relevant for anyone hosting their SAS Viya 4 deployment in Microsoft's Azure Kubernetes Services (AKS). Microsoft Azure's native monitoring, logging and alerting solution is called Azure Monitor. I have recently had a first look at one part of that solution: Azure Monitor Logs.   Over two posts, I'd like to share what I've found so far and hopefully lay a foundation for a more in-depth examination in future. To begin with, in this post we'll see how to create a Log Analytics Workspace from the Azure CLI, how to enable monitoring, and how to reverse both those steps too: disable monitoring and delete the Log Analytics Workspace. They do cost money, after all!   In my next post, we'll use SAS Studio in a SAS Viya 4 deployment on Azure to create some distinctive log messages, and then use Azure Monitor Logs to explore log messages in general and find those distinctive messages. This should serve as a simple introduction to the tools and a starting point from which to explore log analysis in Azure Monitor further.   First, create your Azure AKS cluster   To follow the process of setting up Azure Monitor so that we can play with SAS Viya logs in Azure Monitor Logs, you need a running Azure Kubernetes Service (AKS) cluster. You don't have to set up monitoring before you deploy SAS Viya into that cluster, but it is quite a good idea because the monitoring tools can give you visibility of the progress and health of the deployment process, as well as then allowing you to monitor your SAS Viya 4 deployment too. You can of course also deploy Azure monitoring after deploying SAS Viya 4 in Azure AKS.   I set up my Azure Viya deployment by running some handy scripts in an internal SAS Viya 4 Deployment Workshop. For that reason, some of the variable naming conventions I've adopted below follow patterns used in that workshop. But you could deploy Viya 4 on AKS in any valid way, and the following should work just fine.   If you don't have a Viya 4 deployment running in AKS, be aware that it may take a little time to stand one up. Running the scripts I used, which intentionally provision small host machines, seems to take about 45 minutes to 1 hour.   All the following code examples have a green bar at the side if they are commands you can run (perhaps after modifying them to suit your needs):   Green bars are for commands you can run   The contents of files, expected output of commands and other things that you are not meant to run as commands are shown in boxes with a blue bar at the side:   Blue bars are for files and example command output   Finally, this post includes the code of an example Kusto query from the Microsoft Azure Monitor Logs site, and that is shown in a box with a purple bar:   Purple bars are for Kusto queries Don't worry if the colors are not clear, the words also explain what the content in each box is.   Create a Log Analytics Workspace from the CLI   Note: You should be aware that creating an Azure Log Analytics Workspace incurs a modest cost, in addition to the cost of your Azure AKS cluster. I will assume that the whole process we are about to follow is going to be run from the Azure Command Line Interface (CLI) in a bash shell. Other options are definitely available - not least PowerShell and the Azure Portal web interface.   Begin by opening an Azure CLI shell session, and signing in as a user who has access to the subscription that should be charged for resources you use, i.e. in running the Log Analytics Workspace and in running your Kubernetes cluster.   Like pretty much everything in Azure, your new Log Analytics Workspace must be associated with an Azure Subscription, so that Microsoft knows who to charge for its running costs. Before we create the Log Analytics Workspace, let's list the subscriptions you have access to, from the Azure bash shell prompt:   az account list -o table</pre   If the Azure subscription (let's suppose it is called subscription-name-here) which you intend to have charged for the cost of the Azure Log Analytics Workspace is not currently the default, you can make it the default like this:   az account set -s "subscription-name-here"   Now we can create a text file called deploylaworkspacetemplate.json containing a JSON-format template for your new Log Analytics Workspace resource. The file's contents should be like this:   { "$schema": "https://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "workspaceName": { "type": "String", "metadata": { "description": "Specifies the name of the workspace." } }, "location": { "type": "String", "allowedValues": [ "eastus", "westus" ], "defaultValue": "eastus", "metadata": { "description": "Specifies the location in which to create the workspace." } }, "sku": { "type": "String", "allowedValues": [ "Standalone", "PerNode", "PerGB2018" ], "defaultValue": "PerGB2018", "metadata": { "description": "Specifies the service tier of the workspace: Standalone, PerNode, Per-GB" } } }, "resources": [ { "type": "Microsoft.OperationalInsights/workspaces", "name": "[parameters('workspaceName')]", "apiVersion": "2015-11-01-preview", "location": "[parameters('location')]", "properties": { "sku": { "Name": "[parameters('sku')]" }, "features": { "searchVersion": 1 } } } ] }   Next, set up some environment variables to help us create the Log Analytics Workspace with the right properties.   The first is something we find useful in GEL (the Global Enablement and Learning team in SAS) for a workshop where multiple students share an Azure subscription. We use the STUDENT variable as a prefix in names for other resources. This means students can create their own instance of a resource, with a unique name that is easily identifiable as belonging to them and which are listed together when sorted alphabetically. At the Azure bash shell prompt:   STUDENT=$(az ad signed-in-user show --query mailNickname | sed -e 's/^"//' -e 's/"$//')   In real customer deployments, there are less likely to be lots of workshop students sharing an Azure subscription, but there may still be reason to plan for multiple 'tenant' deployments - different teams in the organization, dev/test/prod deployments etc. may all have their own sets of resources. So, while customers probably wouldn't call it 'STUDENT', they may very well want to set up a variable to identify the 'tenant' team, environment etc., and use it as that variable as one component in otherwise 'standard' resource names.   The other three variables we set up are a bit more self-explanatory, and use our 'tenant' variable (STUDENT) as a prefix in their names so that multiple instances of the same resource will have unique names:   # Set up variables for parameters export RESOURCE_GROUP_NAME=${STUDENT}viya4aks-rg export DEPLOYMENT_NAME=${STUDENT}loganalyticsdeployment export WORKSPACE_NAME=${STUDENT}loganalyticsworkspace Now we are ready to create the Log Analytics Workspace, by running this command:   # Create the Log Analytics Workspace - this can take a few seconds or a few minutes az deployment group create --resource-group $RESOURCE_GROUP_NAME --name $DEPLOYMENT_NAME --template-file /path/to/deploylaworkspacetemplate.json --parameters workspaceName=$WORKSPACE_NAME     Enable Monitoring for your AKS cluster   The command to enable the monitoring add-ons for an AKS cluster takes an argument which passes in the ID of the Log Analytics Workspace to which metrics, logs etc. should be sent. Run this command to get that ID:   LOG_ANALYTICS_WORKSPACE_RESOURCE_ID=`az resource list --name $WORKSPACE_NAME | jq -r '.[] | .id'` echo LOG_ANALYTICS_WORKSPACE_RESOURCE_ID=$LOG_ANALYTICS_WORKSPACE_RESOURCE_ID   Now we can enable monitoring addons for your AKS cluster with name ${STUDENT}viya4aks-aks in the resource group specified in $RESOURCE_GROUP_NAME:   az aks enable-addons --addons monitoring -n ${STUDENT}viya4aks-aks -g $RESOURCE_GROUP_NAME --workspace-resource-id $LOG_ANALYTICS_WORKSPACE_RESOURCE_ID     Check the monitoring agent was deployed to your AKS cluster   Enabling Azure's monitoring addon deploys a daemonset (a type of pod of which one-and-only-one copy runs on a node) called omsagent to each node in your AKS cluster. Let's check that this daemonset is actually present on each node in your cluster. First, see how many nodes you have at the moment (or skip this step if you already know):   # Show active nodes in the Kubernetes cluster at this time kubectl get nodes Here's some example output from the GEL deployment workshop AKS cluster I have been using, which has 5 nodes:   NAME STATUS ROLES AGE VERSION aks-cas-22084866-vmss000000 Ready agent 3h26m v1.18.6 aks-compute-22084866-vmss000000 Ready agent 3h26m v1.18.6 aks-stateful-22084866-vmss000000 Ready agent 3h26m v1.18.6 aks-stateless-22084866-vmss000000 Ready agent 3h26m v1.18.6 aks-system-22084866-vmss000000 Ready agent 3h28m v1.18.6   Use kubectl to check that the 'omsagent' daemonset has been deployed to each node in your AKS Kubernetes cluster, indicating that monitoring is enabled:   # Show how many nodes have the omsagent daemonset deployed kubectl get daemonset omsagent -n kube-system -o wide   Here is the output I get - the omsagent is deployed on all five of my nodes:   NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR omsagent 5 5 5 5 5 <none> 3m44s omsagent mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod08072020 component=oms-agent,tier=node     Confirm log data is being collected in Azure Monitor Logs   To confirm that log data is being sent to the log analytics workspace, open Azure Portal (if it is not already open), navigate to Azure Monitor, and in the pane on the left, choose Logs.   Azure Monitor showing the Logs menu item in the left pane   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   A new logs query should open. Near the top-left corner of the query, click 'Select scope':    In the new query window, click ‘Select scope’     The Select a scope panel appears. Find and select your AKS cluster (a 'Kubernetes service') in this panel. For example, if your username is sukdws, it should be called sukdwsviya4aks-aks.   Tip: A warning message appears near the top of this panel when you select something saying "You may only choose items from the same resource type". Do not worry about this. We will only select one resource, a Kubernetes service, so there is no risk of selecting more than one resource type. Select a scope: choose your AKS cluster, resource type = “Kubernetes service”     When you have selected your AKS cluster (type: Kubernetes service - it's the same thing), click Apply.   Now we'll try one of the example queries to see if we are collecting data. Click the 'Example queries' button, on the right above the New Query window. Select Kubernetes Services from the list on the left, and then scroll down in the panel on the right until you see 'List container logs per namespace':   Azure Monitor Logs example query: List container logs per namespace    Click Run in the box for List container logs per namespace. The example Kusto (KQL) code in this query is:   // List container logs per namespace // View container logs from all the namespaces in the cluster. ContainerLog |join(KubePodInventory| where TimeGenerated > startofday(ago(1h)))//KubePodInventory Contains namespace information on ContainerID |where TimeGenerated > startofday(ago(1h)) | project TimeGenerated ,Namespace , LogEntrySource , LogEntry   The results look something like this:  Results of example query   These are clearly logs from our deployment, but it is not yet easy to see important things like: which container or service generated the log message the severity of the message (DEBUG, INFO, WARNING, ERROR etc.) where there is one the text of the log message   We will look at the query and how to make a better one that presents SAS Viya log data more clearly in my next post.     Cleanup: Disable Monitoring and Remove the Log Analytics Workspace   Azure Monitoring is incurring cost while it collects log data and sends it to your Log Analytics Workspace. To avoid wasting money, you should disable monitoring and delete your Log Analytics Workspace when you have finished with it.   This should disable Monitoring Addons for your AKS cluster with name ${STUDENT}viya4aks-aks, assuming you have set variables with the same values as before for STUDENT and RESOURCE_GROUP_NAME. This may take a couple of minutes:   az aks disable-addons --addons monitoring -n ${STUDENT}viya4aks-aks -g $RESOURCE_GROUP_NAME   That done, again with the same values for the RESOURCE_GROUP_NAME and DEPLOYMENT_NAME variables that you set before, this should delete the Log Analytics Workspace:   az deployment group delete --resource-group $RESOURCE_GROUP_NAME --name $DEPLOYMENT_NAME   Look out for my next post where we'll explore Azure Monitor Logs queries in a bit more detail. As always, please do leave a comment below, and click the 'Like' below if you feel this post has earned it.   To be continued, so see you next time! ... View more

Hi architects,   In case you missed it and are interested, I presented an Ask the Experts webinar last week titled "What Are the Top 10 Tips for SAS® Viya® Administrators?". You can see the recording, slides and the questions asked here: https://communities.sas.com/t5/Ask-the-Expert/What-Are-the-Top-10-Tips-for-SAS-Viya-Administrators-Q-amp-A/ta-p/680405.   Come take a look if that sounds interesting to you.   David ... View more