SAS Viya – CAS Server Life Cycle Management

2 Likes

In SAS® Viya® Platform deployments, the Viya administrator must manage CAS servers. Depending on the maintenance tasks and some operations, the Viya administrator must stop, start, or restart the CAS servers. These lifecycle CAS server operations have impacts on the availability of the CAS data on the users' community.

Since SAS® Viya® stable 2021.2.1 (November 2021), there is a new way to restart the CAS servers that will limit impacts on the users' community.

In this article, I will show you the impacts of stop/start/restart actions on the CAS servers and provide you with information to enable CAS server state transfer.

CAS server lifecycle management: stop/start or restart a CAS server

The CAS server lifecycle is driven by these actions:

Using kubectl delete pods
A CAS server is restarted by deleting all the CAS server pods. The CAS operator pod automatically starts all required CAS server pods just after they were deleted because they are required.

Using kubectl patch casdeployment
- Stop

To scale down a CAS server to zero pods. All CAS server pods are stopped/deleted. The CAS server still exists but without active/running CAS nodes/pods.

- Start

To scale up a CAS serve. All required CAS server pods are started. The CAS operator pod starts all required CAS server pods.

- Restart using the CAS server state transfer capability

A new way to restart a CAS server. State transfer is a process that runs in the background. It creates a new instance of the CAS server and runs until this new instance is fully started and all initial CAS server instance loaded data and existing CAS sessions are transferred.

How it works...

A short video is better to illustrate how all these actions work than a long text...

(view in My Videos)

Pros and Cons - Advantages and disadvantages

	Pros	Cons
Stop/ Start	Very quick to make the CAS server unavailable. Allow the Viya administrator to perform actions before the CAS server start.	Loss of data and sessions. Longer downtime.
Restart	Very quick to clean up the in-memory content and apply configuration changes.	Loss of data and sessions.
State Transfer	Keep the existing loaded data and session active. No downtime for users. Not required to delete pods to update the CAS server.	Requires an extra PVC `sas-cas-transfer-data[-<casInstanceName>]`. Requires more Kubernetes cluster resources. Cost. Not compatible with CAS Auto-Restart after updates. Impact the way to access CAS server pods because their names change.

Notes:

As of SAS® Viya® LST 2022.09, the Viya administrator is not able to stop/start/restart CAS servers using SAS® Environment Manager or the SAS® Viya® Command Line Interface (CLI).

The Viya administrator can only manage the CAS server lifecycle using the Kubernetes CLI: kubectl. This requires that the Viya administrator has elevated privileges in Kubernetes to be able, at least, to:
- Delete CAS pods.
- Be able to patch CAS deployment custom resources.

Enable CAS server state transfer

Enabling CAS state transfer is different depending on which CAS server this must be applied to.

The default CAS server

SAS provided manifests to manage CAS server state transfer for the cas-shared-default CAS server through the Viya deployment assets:

./sas-bases/overlays/cas-server/state-transfer

This overlays directory must be referenced in the Viya deployment kustomization.yaml file into its resources field. This is used to create the additional required PVC: sas-cas-transfer-data[-<casInstanceName>]

./sas-bases/overlays/cas-server/state-transfer/support-state-transfer.yaml

This overlay file must be referenced in the Viya deployment kustomization.yaml file into its transformers field. This is used to enable the CAS server state transfer and mount the additional required PVC

Below is an example of the kustomization.yaml file:

---
namespace: myviyadep

resources:
  - sas-bases/base
  # GEL Specifics to create CA secret for OpenSSL Issuer
  - site-config/security/gel-openssl-ca
  - sas-bases/overlays/network/networking.k8s.io                                         # Using networking.k8s.io API since 2021.1.6
  - site-config/security/openssl-generated-ingress-certificate.yaml                      # Default to OpenSSL Issuer in 2021.2.6
  - sas-bases/overlays/cas-server
  - sas-bases/overlays/internal-postgres
  - site-config/configure-postgres/internal/pgo-client                                   # New Stable 2022.1.3
  - sas-bases/overlays/internal-elasticsearch                                            # New Stable 2020.1.3
  - sas-bases/overlays/update-checker                                                    # added update checker
  - sas-bases/overlays/cas-server/auto-resources                                         # CAS-related
  - sas-bases/overlays/crunchydata_pgadmin                                               # Deploy the sas-crunchy-data-pgadmin container
  - site-config/sas-prepull/add-prepull-cr-crb.yaml
  - sas-bases/overlays/cas-server/state-transfer                                         # Enable state transfer for the cas-shared-default CAS server - new PVC sas-cas-transfer-data

configurations:
  - sas-bases/overlays/required/kustomizeconfig.yaml

transformers:
  - sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml                    # New Stable 2020.1.3
  - sas-bases/overlays/startup/ordered-startup-transformer.yaml
  - sas-bases/overlays/required/transformers.yaml
  - site-config/mirror.yaml
  - site-config/daily_update_check.yaml                                                  # change the frequency of the update-check
  - sas-bases/overlays/cas-server/auto-resources/remove-resources.yaml                   # CAS-related
  - sas-bases/overlays/internal-elasticsearch/internal-elasticsearch-transformer.yaml    # New Stable 2020.1.3
  - sas-bases/overlays/sas-programming-environment/enable-admin-script-access.yaml       # To enable admin scripts
  #- sas-bases/overlays/scaling/zero-scale/phase-0-transformer.yaml
  #- sas-bases/overlays/scaling/zero-scale/phase-1-transformer.yaml
  - sas-bases/overlays/cas-server/state-transfer/support-state-transfer.yaml             # Enable state transfer for the cas-shared-default CAS server - enable and mount new PVC

components:
  - sas-bases/components/security/core/base/full-stack-tls
  - sas-bases/components/security/network/networking.k8s.io/ingress/nginx.ingress.kubernetes.io/full-stack-tls

patches:
  - path: site-config/storageclass.yaml
    target:
      kind: PersistentVolumeClaim
      annotationSelector: sas.com/component-name in (sas-backup-job,sas-data-quality-services,sas-commonfiles,sas-cas-operator,sas-pyconfig)

secretGenerator:
  - name: sas-consul-config
    behavior: merge
    files:
      - SITEDEFAULT_CONF=site-config/sitedefault.yaml
  - name: sas-image-pull-secrets
    behavior: replace
    type: kubernetes.io/dockerconfigjson
    files:
      - .dockerconfigjson=site-config/crcache-image-pull-secrets.json

configMapGenerator:
  - name: ingress-input
    behavior: merge
    literals:
      - INGRESS_HOST=myviyadep.myserver.race.sas.com
  - name: sas-shared-config
    behavior: merge
    literals:
      - SAS_SERVICES_URL=https://myviyadep.myserver.race.sas.com
  - name: input
    behavior: merge
    literals:
      - IMAGE_REGISTRY=crcache-race-sas-cary.unx.sas.com

Then, generate the SASDeployment Custom Resource, or the site.yaml manifest, and apply it.

All other CAS servers

The create-cas-server.sh script is provided by SAS into the SAS Viya deployment asset. This script is used by the Viya administrator to generate manifests for additional CAS servers.

[myuser@myserver myviyadep]$ bash ./sas-bases/examples/cas/create/create-cas-server.sh --help
Flags:
  -h  --help     help
  -i, --instance CAS server instance name
  -o, --output   Output location. If undefined, default to working directory.
  -v, --version  CAS server creation utility version
  -w, --workers  Specify the number of CAS worker nodes. Default is 0 (SMP).
  -b, --backup   Set this to include a CAS backup controller. Disabled by default.
  -t, --tenant   Set the tenant name. default is shared.
  -r, --transfer Set this to enable support for state transfer between restarts. Disabled by default.
  -a, --affinity Specify the node affinity and toleration to use for this deployment.  Default is 'cas'.
  -q, --required-affinity Set this flag to have the node affinity be a required node affinity.  Default is preferred node affinity.

For the non default (cas-shared-default) CAS servers, the state transfer is enabled by using the “-r, --transfer” option of the create-cas-server.sh script

bash ./sas-bases/examples/cas/create/create-cas-server.sh --instance newcasserver --output ./site-config --workers 4 --backup 0 --transfer 1

Note: This command will generate the manifests for a CAS server that will be named cas-shared-newcasserver. This CAS server will be massively parallel processing (MPP) with four workers, and the state transfer will be enabled. Here is the command output:

ri Jan 13 09:43:33 EST 2023 - instance = newcasserver
Fri Jan 13 09:43:33 EST 2023 - tenant =
Fri Jan 13 09:43:33 EST 2023 - output = ./site-config
output directory does not exist: ./site-config/
creating directory: ./site-config/
Generating artifacts...
100.0% [=======================================================================]
|-cas-shared-newcasserver (root directory)
  |-cas-shared-newcasserver-cr.yaml
  |-kustomization.yaml
  |-shared-newcasserver-pvc.yaml
  |-annotations.yaml
  |-backup-agent-patch.yaml
  |-cas-consul-sidecar.yaml
  |-cas-fsgroup-security-context.yaml
  |-cas-sssd-sidecar.yaml
  |-kustomizeconfig.yaml
  |-provider-pvc.yaml
  |-transfer-pvc.yaml
  |-enable-binary-port.yaml
  |-enable-http-port.yaml
  |-configmaps.yaml
  |-state-transfer.yaml
  |-node-affinity.yaml
  |-require-affinity.yaml

create-cas-server.sh complete!

The create-cas-server.sh script generates all manifests for the CAS server, even if not used. The create-cas-server.sh script options are used to set some values into the manifests and used to reference these manifests into the CAS server kustomization.yaml file.

When enabling the state transfer, transfer-pvc.yaml is referenced into the resources field and state-transfer.yaml into the transformers field of the CAS server kustomization.yaml file.

Ensure that this CAS server overlays directory is referenced in the Viya deployment kustomization.yaml file into its resources field. Here is an example of the kustomization.yaml file

---
namespace: myviyadep

resources:
  - sas-bases/base
  # GEL Specifics to create CA secret for OpenSSL Issuer
  - site-config/security/gel-openssl-ca
  - sas-bases/overlays/network/networking.k8s.io                                         # Using networking.k8s.io API since 2021.1.6
  - site-config/security/openssl-generated-ingress-certificate.yaml                      # Default to OpenSSL Issuer in 2021.2.6
  - sas-bases/overlays/cas-server
  - sas-bases/overlays/internal-postgres
  - site-config/configure-postgres/internal/pgo-client                                   # New Stable 2022.1.3
  - sas-bases/overlays/internal-elasticsearch                                            # New Stable 2020.1.3
  - sas-bases/overlays/update-checker                                                    # added update checker
  - sas-bases/overlays/cas-server/auto-resources                                         # CAS-related
  - sas-bases/overlays/crunchydata_pgadmin                                               # Deploy the sas-crunchy-data-pgadmin container
  - site-config/sas-prepull/add-prepull-cr-crb.yaml
  - sas-bases/overlays/cas-server/state-transfer                                         # Enable state transfer for the cas-shared-default CAS server - new PVC sas-cas-transfer-data
  - site-config/cas-shared-newcasserver                                                  # To add a new CAS server
configurations:
  - sas-bases/overlays/required/kustomizeconfig.yaml

transformers
  - sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml                    # New Stable 2020.1.3
  - sas-bases/overlays/startup/ordered-startup-transformer.yaml
  - sas-bases/overlays/required/transformers.yaml
  - site-config/mirror.yaml
  - site-config/daily_update_check.yaml                                                  # change the frequency of the update-check
  - sas-bases/overlays/cas-server/auto-resources/remove-resources.yaml                   # CAS-related
  - sas-bases/overlays/internal-elasticsearch/internal-elasticsearch-transformer.yaml    # New Stable 2020.1.3
  - sas-bases/overlays/sas-programming-environment/enable-admin-script-access.yaml       # To enable admin scripts
  #- sas-bases/overlays/scaling/zero-scale/phase-0-transformer.yaml
  #- sas-bases/overlays/scaling/zero-scale/phase-1-transformer.yaml
  - sas-bases/overlays/cas-server/state-transfer/support-state-transfer.yaml             # Enable state transfer for the cas-shared-default CAS server - enable and mount new PVC

components:
  - sas-bases/components/security/core/base/full-stack-tls
  - sas-bases/components/security/network/networking.k8s.io/ingress/nginx.ingress.kubernetes.io/full-stack-tls

patches:
  - path: site-config/storageclass.yaml
    target:
      kind: PersistentVolumeClaim
      annotationSelector: sas.com/component-name in (sas-backup-job,sas-data-quality-services,sas-commonfiles,sas-cas-operator,sas-pyconfig)

secretGenerator:
  - name: sas-consul-config
    behavior: merge
    files:
      - SITEDEFAULT_CONF=site-config/sitedefault.yaml
  - name: sas-image-pull-secrets
    behavior: replace
    type: kubernetes.io/dockerconfigjson
    files:
      - .dockerconfigjson=site-config/crcache-image-pull-secrets.json

configMapGenerator:
  - name: ingress-input
    behavior: merge
    literals:
      - INGRESS_HOST=myviyadep.myserver.race.sas.com
  - name: sas-shared-config
    behavior: merge
    literals:
      - SAS_SERVICES_URL=https://myviyadep.myserver.race.sas.com
  - name: input
    behavior: merge
    literals:
      - IMAGE_REGISTRY=crcache-race-sas-cary.unx.sas.com

Then, generate the SASDeployment Custom Resource, or the site.yaml manifest, and apply it.

Impacts of CAS server state transfer

Enabling CAS server state transfer is not free. This new capability has impacts on the way that CAS servers are managed, this has an impact on the Kubernetes cluster resources that are used by the CAS servers, and the way that Viya administrator could have to interact with the CAS servers.

An additional PVC is required to operate the CAS server.

This additional PVC needs to be big enought to accommodate the copy of all data that are loaded in memory (global data and session data) and all sessions that are running when the state transfer is initiated.

CAS server requires the double of resources to be operate.

The double of Kubernetes cluster resources (CPU, Memory, Disk, Kubernetes nodes) used by the CAS server is required during the full state transfer process. These extra resources are released at the end of the CAS server state transfer process.

- During the CAS server operation, it uses some resources in the Kubernetes cluster.
  
  Select any image to see a larger version.
  Mobile users: To view the images, select the "Full" version at the bottom of the page.

- During the CAS server state transfer, it use two time this number of resources until the state transfer is finalized.

This resource impact is more important if the CAS server AUTORESOURCES mode is used in the SAS® Viya® Platform deployment. This requires having a fully dedicated Kubernetes node for each CAS server pod.

- When the CAS server state transfer is fully completed, the resources used by the previous instance of the CAS server are released.

CAS server pods name change based on the casoperator.sas.com/instance-index value. Each time the state transfer is initiated for a CAS server, the CAS server instance-index value is incremented.

If you have scripts that require to access CAS server pods, it is required to modify these scripts to take care of the CAS server new pods names that use the casoperator.sas.com/instance-index value if this value is not '0'.

Note that the casoperator.sas.com/instance-index value is reset to '0' only when the CAS server is stopped or its pods are deleted. In this case, the new CAS server pods will start with all initial labels' values.

Which technique? When? Why?

These different ways to stop/start/restart a CAS server are tools that knid be used by the Viya administrator to manage the SAS® Viya® Platform deployments CAS servers depending the context. They all have specific interests.

Stop then start

The quickest and best way to remove a CAS server from the SAS® Viya® Platform deployments. Specifically, when the Viya administrator wants to make this CAS server available again later.

Restart using the kubectl delete pods
- The quickest way to reinitialize a CAS server in case of issues.
- The way to reinitialize a CAS server instance index during a maintenance window (To do not impact the users' community), when the CAS server state transfer is enabled.

Restart using the CAS server state transfer capability

The best way to maintain the SAS® Viya® Platform deployments or manage the CAS server topology changes without impacting the users' community.

Note: Be careful, if the SAS® Viya® Platform deployments maintenance requires changes on the CAS server, it could be required to regenerate the CAS server manifests. This implies that the CAS server state transfer could not be used.

I hope this article has been helpful to you.

References:

SAS documentation

SAS® Viya® Administration

SAS® Viya® Operations
- Deployment / Installation / Common Customizations / Configure CAS Settings / Enable State Transfer f...

Find more articles from SAS Global Enablement and Learning here.