Limit a user’s simultaneous compute server processes in SAS Viya 2021.1 and later

4 Likes

A feature in the SAS Launcher which limits the number of concurrent compute sessions for each user was added in SAS Viya 2021.1. The idea is to provide denial-of-service protection by preventing users from maliciously creating a large number of compute sessions. But it also provides a feature that SAS Administrators have long asked for, to help them manage fair use of compute resources.

In this post, we'll see how it works in practice, how to tell what your current limits are, and how to change the limits or remove them completely.

SAS Launcher's User Process Limits

Let's suppose a user called Henrik has already started all the compute servers he is allowed to, by opening multiple SAS Studio sessions in separate browser tabs. Each of them started as normal, and each has a SAS Compute server process running under the SAS Studio compute context:

A SAS session has been started successfully in the SAS Studio compute context.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Then Henrik, who is not being very considerate of other system users, opens SAS Studio in yet another browser tab, and waits while the button top right displays a wait symbol indicating that it is attempting to start another compute server for him:

Waiting for a SAS Compute Server session to start.

After a much longer wait than for the previous successful sessions, SAS Studio displays a red 'x' in a circle, indicating that the compute server failed to start. If Henrik hovers his pointer over the button, a tooltip gives only a minimal hint about what is wrong: "An attempt to launch a SAS session has failed":

Failed to launch a SAS session in the SAS Studio compute context.

Unless Henrik has access to the SAS Studio web application logs (see below), this is all the information Henrik gets about why his latest compute session failed to start, when many others previously started normally. He may learn than he can start only so many sessions at a time, but it would be helpful for the SAS Administrator to let users know there is a limit, and what that limit is.

If Henrik then closes one of his existing compute sessions, so that the compute server pod which hosts that session terminates, he will find that he is able to open a new compute session again.

See the real error message

In whatever logging solution is available to the administrator (here we are using Kibana from the SAS Viya Monitoring for Kubernetes project), the administrator can search for log messages from logsource: sas-studio-app and level: ERROR, to find a more informative error message explaining why the compute session did not start for Henrik:

We can find a more informative error message in Kibana

The full error message reads:

com.sas.studio.exceptions.ServerOperationException: Unable to create compute server session. Unable to complete the launch request, max number of active processes has been reached for the user: user=Henrik limit=2

So while the user session limit is n, the reason for a failure to launch a user's SAS session n+1 may not be immediately obvious to that user in SAS Studio. But it is something the administrator can explain.

See the current limits on SAS Compute Sessions

As a Kubernetes administrator, or at least a user with access to the Kubernetes cluster and permission to describe the SAS Launcher deployment in the SAS Viya namespace, you can see the current user process limit by running this kubectl command:

kubectl describe deploy sas-launcher | grep SAS_LAUNCHER_USER_PROCESS_LIMIT

Expected output in an environment where the default values have not been changed:

      SAS_LAUNCHER_USER_PROCESS_LIMIT:                10
      SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED:        true

By default users are allowed to have 10, 25 or 50 simultaneous SAS Compute processes running at a time, each in their own separate Kubernetes pod. The limit depends on the deployment size per the table below.

There is also a new custom group called LauncherSuperUsers, empty by default. It allows the administrator to define a separate limit for users who e.g. need more sessions. For users who are directly or indirectly members of this custom group, the SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT limit applies instead. To see all four values, run:

kubectl describe deploy sas-launcher | grep SAS_LAUNCHER.*USER_PROCESS_LIMIT

Expected output:

SAS_LAUNCHER_USER_PROCESS_LIMIT: 10
SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED: true
SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT: 15
SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT_ENABLED: true

The USER and SUPER_USER limits can be disabled by setting SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED and/or SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT_ENABLED to false. The default size of the two limits depends on the size of your deployment - we can infer from this that my SAS Viya deployment is a 'Small' deployment:

User limits for compute sessions by deployment size

How to change SAS Launcher User Process Limits

We have seen that the SAS Launcher user process limits are set by four environment variables set for the Kubernetes deployment for sas-launcher. Here is how the SAS Viya Administration documentation explains them:

Variable	Description
SAS_LAUNCHER_USER_PROCESS_LIMIT	Default: 10 (Small), 25 (Medium), 50 (Large) Modifies the process limit by users who do not belong to the LauncherSuperUsers custom group.
SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED	Default: TRUE Disables or enables the process limit for users who do not belong the LauncherSuperUsers custom group. If the value is FALSE (disabled), then the user can have an unlimited number of active processes running.
SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT	Default: 15 (Small), 35 (Medium), 65 (Large) Modifies the process limit for users who belong to the LauncherSuperUsers custom group.
SAS_LAUNCHER_SUPER_USER_PROCESS_LIMIT_ENABLED	Default: TRUE Disables or enables the process limit for users who belong to the LauncherSuperUsers custom group. If the value is FALSE (disabled), then the user can have an unlimited number of active processes running.

So, how can we change them?

Quick temporary method

You can temporarily change the values using kubectl. The example below sets SAS_LAUNCHER_USER_PROCESS_LIMIT to 2, but you can set any of the other environment variables the same way:

kubectl set env deploy sas-launcher SAS_LAUNCHER_USER_PROCESS_LIMIT=2

Note that this command will update the value in the deployment now, but it won't change the SAS Viya deployment manifest - usually called site.yaml - that was used and may well be used again to deploy SAS Viya into the cluster. Any changes to Kubernetes environment variables you apply like this would likely be lost next time site.yaml is applied to the cluster.

Longer but permanent method

To set these SAS Launcher environment variables more permanently, add a patchTransformer to your kustomization.yaml file, re-run kustomize to re-generate your site.yaml file, and re-apply it to your Kubernetes cluster. We'll create a separate patchTransformer yaml file, and then reference it from our kustomization.yaml, in the same way as many other Kubernetes-level configuration changes are made to SAS Viya deployments.

If your SAS Viya deployment directory is at /home/cloud-user/project/deploy/ and the deployment is called gelcorp, you can create the patchTransformer yaml file something like this:

tee /home/cloud-user/project/deploy/gelcorp/site-config/launcher-user-process-limit.yaml  > /dev/null << EOF
###############################################################################
# Kustomize patch configuration to change the process limit per user for
# sas-launcher.
# See README.md for more information.
###############################################################################
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-launcher-processes-per-user-limitation
patch: |-
  - op: replace
    path: /spec/template/spec/containers/0/env/1
    value:
      name: SAS_LAUNCHER_USER_PROCESS_LIMIT
      value: "2"
      # value: "25"
      # value: "50"
target:
  version: v1
  group: apps
  kind: Deployment
  name: sas-launcher
EOF

That done, reference it in the kustomization.yaml file by adding a line like this, if it is not already present, in the transformers section:

- site-config/launcher-user-process-limit.yaml

Here's a bit of script that may do that for you, using grep and yq:

_FILE=~/project/deploy/gelcorp/kustomization.yaml
_MATCH_LINE="site-config/launcher-user-process-limit.yaml"

if grep -qe "$_MATCH_LINE" $_FILE ; then
  echo "NOTE: Found \""${_MATCH_LINE}"\" in \""${_FILE}"\". We should not insert it again." ;
else
  echo "NOTE: Did not find \""${_MATCH_LINE}"\" in \""${_FILE}"\". Adding it." ;
  printf "
  - command: update
    path: transformers[+]
    value:
      ${_MATCH_LINE}
  " | yq -I 4 w -i -s - $_FILE
fi

Here's part of my kustomization.yaml file containing the inserted line:

.
.
.
configurations:
- sas-bases/overlays/required/kustomizeconfig.yaml
transformers:
- sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml # TLS
.
.
.
- site-config/launcher-user-process-limit.yaml
patches:
.
.
.

Then, when you are ready, rebuild site.yaml and apply it to your cluster in the usual way, with commands something like:

kustomize build -o site.yaml then kubectl apply -f site.yaml

When you have done that, you should see that the value of SAS_LAUNCHER_USER_PROCESS_LIMIT has changed to whatever value you set it to. Do demonstrate the effect for this blog post, I set it to the unusually low value of 2, so that's what I see when I re-run:

kubectl describe deploy sas-launcher | grep SAS_LAUNCHER_USER_PROCESS_LIMIT

Expected results after applying the change by either of the above two methods:

      SAS_LAUNCHER_USER_PROCESS_LIMIT:                2
      SAS_LAUNCHER_USER_PROCESS_LIMIT_ENABLED:        true

Restore the original value

If you changed the values using the quick temporary method, using kubectl set env, you can either run another similar command to restore the variables you changed to their previous values, or you could re-apply your site.yaml file to do so.

If you changed the values by adding a reference to a patchTransformer to your kustomize.yaml file, restore the default value by removing or commenting the line referencing site-config/launcher-user-process-limit.yaml from your kustomization.yaml and repeating the steps above to re-create and reapply site.yaml.

Reference

See the documentation for this feature in the SAS Viya Administration guide > Servers and Services > Programming Run-Time Environment > Programming Run-Time Servers > SAS Launcher Service > Configure Denial-of-Service Protection for detail: a https://go.documentation.sas.com/doc/en/sasadmincdc/v_014/calsrvpgm/n00002viyaprgmsrvs00000admin.htm...

See you next time!