Workload Management from the Command Line
Views
2,745
Workload Management on SAS Viya is here. You've read about considerations for installers and architects, and about the evolution of Workload Management, but there's also good news for administrators, who can make use of several command-line tools to manage and monitor workloads and configuration.
Frederik Vandenberghe (@FrederikV) also authored a great post about how various components and services interact with each other to make the Workload Management magic happen. As Frederik points out, there are different ways to submit jobs to the Workload Orchestrator. For example, from SAS Studio, via SAS/CONNECT, from the command line - in fact, pretty much all 'compute' workloads run in the 'grid' in a deployment that includes Workload Management in the SAS license.
We'll begin by demonstrating the submission of a job from the command line, but before we do that, remember that the term 'job' can mean many things in SAS Viya. Although it's possible to create and schedule job requests and flows from the command line, we'll keep things simple and submit a simple job using the sas-viya CLI's batch plugin.
/opt/sas/viya/home/bin/sas-viya batch jobs submit-pgm -c priority --pgm /tmp/doWork2mins.sas --restart
>>> File set created, ID="JOB_20220207_085120_784_1"
>>> Uploading "doWork2mins.sas"
>>> Job submitted, ID="859ac9ef-371e-478a-a908-a3ecc9a881d9", Workload Job ID="91".
The doWork2mins program, which is stored locally, is first uploaded and then executed in a launcher session. In this case, we've also specified the batch context we'd like the job to run in. Batch and Launcher contexts can be associated with Workload Management queues, so jobs can be targeted to specific queues as required.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
While the job is running, we can see what it's doing by running:
/opt/sas/viya/home/bin/sas-viya --output text batch jobs list --job-id=859ac9ef-371e-478a-a908-a3ecc9a881d9
ID Name Workload Job ID Created By State Submitted Time Started Time Ended Time Return Code
859ac9ef-371e-478a-a908-a3ecc9a881d9 doWork2mins 91 geladm running 2022-02-07T03:51:22 2022-02-07T03:51:22 2022-02-07T03:53:34 0
View all jobs by removing the --job-id flag. Optionally, we can sort or filter the output (using REST API filter query parameters), and we can add the --details flag to display more information, such as the execution host, in the output:
/opt/sas/viya/home/bin/sas-viya batch jobs list --details
ID Name Workload Job ID Created By Context ID File Set ID Process ID State Execution Host Return Code Submitted Time Started Time Ended Time Creation Time Modified By Modified Time
...
859ac9ef-371e-478a-a908-a3ecc9a881d9 doWork2mins 91 geladm 43591c29-8a83-4893-8313-8e6dc1f9f63d JOB_20220207_085120_784_1 9f9e7655-9e6c-478d-a583-fe64ed24d16b completed 10.42.3.99 0 2022-02-07T03:51:22 2022-02-07T03:51:22 2022-02-07T03:53:34 2022-02-07T03:51:22 2022-02-07T03:53:39
c7b8413d-d991-4330-ba79-822669c2b7ca doWork5mins 90 geladm 43591c29-8a83-4893-8313-8e6dc1f9f63d JOB_20220128_043256_593_1 f9c4c017-ce22-402d-bab9-6df70eb9f20c completed 10.42.4.89 0 2022-01-27T23:32:58 2022-01-27T23:32:58 2022-01-27T23:38:12 2022-01-27T23:32:58 2022-01-27T23:38:16
b72eceb0-fdde-4f23-a126-73f05e97e385 doWork2mins 92 geladm 43591c29-8a83-4893-8313-8e6dc1f9f63d JOB_20220207_085338_802_1 9ec536e4-7870-4a1d-b8ab-b9258a506a92 completed 10.42.4.90 0 2022-02-07T03:53:39 2022-02-07T03:53:39 2022-02-07T03:55:54 2022-02-07T03:53:39 2022-02-07T03:55:58
That's great for running jobs in isolation, but what about flows? We can create flows using the jobs CLI plug-in (or from the Jobs and Flows page in Environment Manager), but as it stands today, all jobs in a flow will run in the Job Execution launcher context, which means that all jobs executed/scheduled this way will be directed to the (same) queue the context is associated with. New features and enhancements are planned for flows in future, so watch this space for updates on this.
Now we've seen that we can use the batch and jobs plug-ins to the sas-viya CLI, but both were available even before the release of Workload Management. So what's new now? Enter the workload-orchestrator plugin. Using this new plug-in, we can view and interact with workloads, and modify the configuration of queues, hosts, logs and more. Everything that can be done from the Workload Orchestrator page in Environment Manager can also be done with this CLI plug-in.
For example, we can see jobs running in a queue (pro tip: JSON output is a little easier to read with this plug-in):
/opt/sas/viya/home/bin/sas-viya workload-orchestrator jobs list -queue priority
queue: priority
no job id specified, returning information for all jobs.
collecting output...
{
"jobs": [
{
"id": 93,
"processingInfo": {
"consumedResources": [
{
"name": "cores",
"value": 0.15
},
{
"name": "memory",
"value": 536.102295
}
],
"endTime": "",
"exchangeCreds": false,
"executionHost": "intnode04",
"executionHostAliases": [
"sas-workload-orchestrator-9qfwk"
],
"exitCode": 0,
"hosts": [],
"k8sObjName": "sas-launcher-ec306923-6e67-4991-9ca2-13bff95e7a09-93",
"lastCheckTime": "2022-02-07T10:15:58Z",
"launchStatus": 0,
"limitValues": [
{
"name": "maxMemory",
"value": 0
},
{
"name": "maxCpuTime",
"value": 0
},
{
"name": "maxClockTime",
"value": 0
},
{
"name": "maxIoTotal",
"value": 0
},
{
"name": "maxIoRate",
"value": 0
},
{
"name": "maxPageRate",
"value": 0
}
],
"limits": [],
"processId": 0,
"queue": "priority",
"queuePriority": 100,
"request": {
"args": null,
"cmd": "",
"consumedResources": null,
"envVars": null,
"exchangeCreds": false,
"hosts": null,
"k8sJobDef": "",
"k8sNamespace": "",
"k8sSecrets": null,
"kvPairs": null,
"limits": null,
"name": "",
"project": "",
"queue": "",
"requiredResources": null,
"requiredTags": null,
"restartable": false,
"startAfter": "",
"type": ""
},
"requiredResources": [],
"requiredTags": [],
"startTime": "2022-02-07T10:15:58Z",
"state": "STARTING",
"submitTime": "2022-02-07T10:15:58Z",
"suspendFlags": 0,
"totalTimePending": 0,
"totalTimeRunning": 0,
"totalTimeSuspAdmin": 0,
"totalTimeSuspPreempt": 0,
"totalTimeSuspThresh": 0,
"userGroup": ""
},
"request": {
"args": [],
"cmd": "",
"consumedResources": [
{
"name": "cores",
"value": 0.15
},
{
"name": "memory",
"value": 536.102295
}
],
"envVars": [],
"exchangeCreds": false,
"hosts": [],
"k8sJobDef": "{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"labels\":{\"sas-launcher-job-container-name\":\"sas-programming-environment\"},\"name\":\"sas-launcher-ec306923-6e67-4991-9ca2-13bff95e7a09\"},\"spec\":{\"backoffLimit\":6,\"completions\":1,\"parallelism\":1,\"template\":{\"metadata\":{\"annotations\":{\"sas.com/certificate-file-format\":\"pem\",\"sas.com/kustomize-base\":\"sas-programming-environment\",\"sas.com/tls-enabled-ports\":\"all\",\"sas.com/tls-mode\":\"full-stack\",\"seccomp.security.alpha.kubernetes.io/pod\":\"runtime/default\",\"sidecar.istio.io/inject\":\"false\",\"sas.com/owner\":\"4000\",\"sas.com/ownerGroup\":\"2002\"},\"labels\":{\"launcher.sas.com/job-type\":\"sas-batch-job\",\"sas.com/created-by\":\"sas-launcher\",\"launcher.sas.com/username\":\"geladm\",\"launcher.sas.com/requested-by-client\":\"sas.cli\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"preference\":{\"matchExpressions\":[{\"key\":\"workload.sas.com/class\",\"operator\":\"In\",\"values\":[\"compute\"]}]},\"weight\":100},{\"preference\":{\"matchExpressions\":[{\"key\":\"workload.sas.com/class\",\"operator\":\"NotIn\",\"values\":[\"cas\",\"connect\",\"stateless\",\"stateful\"]}]},\"weight\":50}],\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"kubernetes.azure.com/mode\",\"operator\":\"NotIn\",\"values\":[\"system\"]}]}]}},\"podAffinity\":{},\"podAntiAffinity\":{}},\"automountServiceAccountToken\":true,\"containers\":[{\"command\":[\"/opt/sas/viya/config/etc/sasgrid/default/sasgrid\",\"SASBATCHSASPGM:'doWork2mins.sas'\",\"SASBATCHWRKDIR:'JOB_20220207_101557_144_1'\",\"SASCHKRESTART:'1'\",\"SASCOMMANDARG:\"],\"env\":[{\"name\":\"SAS_K8S_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.name\"}}},{\"name\":\"SAS_K8S_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"SAS_LICENSE\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"SAS_LICENSE\",\"name\":\"sas-license-k4kf9kt488\"}}},{\"name\":\"SAS_URL_SERVICE_TEMPLATE\",\"valueFrom\":{\"configMapKeyRef\":{\"key\":\"SAS_URL_SERVICE_TEMPLATE\",\"name\":\"sas-shared-config-k454gfd685\"}}},{\"name\":\"SASCLOUDNATIVE\",\"value\":\"1\"},{\"name\":\"HOSTIP\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"status.podIP\"}}},{\"name\":\"SASREDIRSASOPTS\",\"value\":\"-noauth\"},{\"name\":\"SASREDIRCMDOPTS\",\"value\":\"-addshell -noauth\"},{\"name\":\"SAS_TLS_LISTENER_ENABLED\",\"value\":\"true\"},{\"name\":\"SAS_LAUNCHER_SERVER_OS\",\"value\":\"linux\"},{\"name\":\"SAS_CADENCE_RELEASE\",\"value\":\"20220117.1642385447408\"},{\"name\":\"SAS_DEPLOYMENT_ID\",\"value\":\"viya\"},{\"name\":\"SAS_BUILD_TYPE\",\"value\":\"x64_oci_linux_2-docker\"},{\"name\":\"SAS_ALLOW_ADMIN_SCRIPTS\",\"value\":\"true\"},{\"name\":\"SAS_DEPLOYMENT_TYPE\",\"value\":\"default\"},{\"name\":\"SAS_SPRE_VAR_PATH_LOG\",\"value\":\"log\"},{\"name\":\"ODBCINST\",\"value\":\"/opt/sas/viya/home/lib64/accessclients/odbcinst.ini\"},{\"name\":\"SAS_LAUNCHER_PROCESS_ID\",\"value\":\"ec306923-6e67-4991-9ca2-13bff95e7a09\"},{\"name\":\"SAS_REPOSITORY_WAREHOUSE_URL\",\"value\":\"https://ses.sas.download/ses\"},{\"name\":\"VIYA_LOCKDOWN_DEFAULT_METHODS\",\"value\":\"http email ftp hadoop java\"},{\"name\":\"SAS_PATH_SEPARATOR\",\"value\":\"/\"},{\"name\":\"SAS_SERVICES_URL\",\"value\":\"https://gelcorp.rext03-0142.race.sas.com\"},{\"name\":\"SAS_LAUNCHER_CONSUL_URL\",\"value\":\"https://sas-consul-server:8500\"},{\"name\":\"SAS_SPRE_VAR_PATH\",\"value\":\"/opt/sas/viya/config/var\"},{\"name\":\"ODBCSYSINI\",\"value\":\"/opt/sas/viya/home/lib64/accessclients/\"},{\"name\":\"SAS_CADENCE_DISPLAY_NAME\",\"value\":\"Long-Term Support 2021.2\"},{\"name\":\"SAS_BASE_CADENCE_NAME\",\"value\":\"stable\"},{\"name\":\"CAS_CLIENT_SSL_CA_LIST\",\"value\":\"/security/trustedcerts.pem\"},{\"name\":\"JDBCDRIVERS\",\"value\":\"/opt/sas/viya/home/lib64/accessclients/jdbc\"},{\"name\":\"ODBCHOME\",\"value\":\"/opt/sas/viya/home/lib64/accessclients\"},{\"name\":\"SAS_LAUNCHER_DEPLOYMENT_CONFIG_INSTANCE_ID\",\"value\":\"default\"},{\"name\":\"SAS_EXT_LLP_ACCESS\",\"value\":\"/opt/sas/viya/home/lib64/accessclients/lib:/opt/sas/viya/home/lib64\"},{\"name\":\"POSTGRES\",\"value\":\"/opt/sas/viya/home/lib64\"},{\"name\":\"SAS_LAUNCHER_DEPLOYMENT_CONFIG_PATH\",\"value\":\"/opt/sas/viya/config\"},{\"name\":\"SAS_BASE_CADENCE_VERSION\",\"value\":\"2021.1.6\"},{\"name\":\"SAS_SERVER_OS\",\"value\":\"linux\"},{\"name\":\"SAS_LAUNCHER_INIT_ENCODING_DEFAULT\",\"value\":\"utf8\"},{\"name\":\"SAS_CERTIFICATE_FILE\",\"value\":\"/security/tls.crt\"},{\"name\":\"SAS_TRUSTED_CA_CERTIFICATES_JKS_FILE\",\"value\":\"/security/trustedcerts.jks\"},{\"name\":\"SAS_LAUNCHER_SAS_SERVICES_URL\",\"value\":\"https://gelcorp.rext03-0142.race.sas.com\"},{\"name\":\"SAS_SPRE_APP_COMPUTE\",\"value\":\"compsrv\"},{\"name\":\"SAS_SPRE_APP_BATCH\",\"value\":\"batch\"},{\"name\":\"SAS_SPRE_VAR_PATH_RUN\",\"value\":\"run\"},{\"name\":\"SAS_DEPLOYMENT_INSTALL_PATH\",\"value\":\"/opt/sas/viya/home\"},{\"name\":\"SAS_CADENCE_DISPLAY_VERSION\",\"value\":\"2021.2\"},{\"name\":\"VIYA_LOCKDOWN_ENABLE\",\"value\":\"1\"},{\"name\":\"SAS_CLIENT_TOKEN\",\"value\":\"eyJhbGciOiJSUzI1NiIsImprdSI6Imh0dHBzOi8vbG9jYWxob3N0L1NBU0xvZ29uL3Rva2VuX2tleXMiLCJraWQiOiJsZWdhY3ktdG9rZW4ta2V5IiwidHlwIjoiSldUIn0.eyJqdGkiOiI1YTA1Mzc1MmVjMTQ0MTczYjU0NWU5MmE0YTZkNjE5ZiIsImV4dF9pZCI6InVpZD1nZWxhZG0sb3U9dXNlcnMsZGM9Z2VscmFjZSxkYz1jb20iLCJzdWIiOiI5OGEzZWE0ZS00YjUyLTQwZDEtYTRjYy0wMjhiYjQ1MjQyMmYiLCJzY29wZSI6WyJzYXN1c2VycyIsIkhSIiwiU2FsZXMiLCJwb3dlcnVzZXJzIiwic2FzYWRtaW5zIiwiY2xpZW50cy5yZWFkIiwiY2xpZW50cy5zZWNyZXQiLCJ1YWEucmVzb3VyY2UiLCJGaW5hbmNlIiwib3BlbmlkIiwidWFhLmFkbWluIiwiY2xpZW50cy5hZG1pbiIsInNjaW0ucmVhZCIsIlNBU0FkbWluaXN0cmF0b3JzIiwiY2xpZW50cy53cml0ZSIsIkdFTENvcnBTeXN0ZW1BZG1pbnMiLCJzY2ltLndyaXRlIl0sImNsaWVudF9pZCI6InNhcy5jbGkiLCJjaWQiOiJzYXMuY2xpIiwiYXpwIjoic2FzLmNsaSIsImdyYW50X3R5cGUiOiJwYXNzd29yZCIsInVzZXJfaWQiOiI5OGEzZWE0ZS00YjUyLTQwZDEtYTRjYy0wMjhiYjQ1MjQyMmYiLCJvcmlnaW4iOiJsZGFwIiwidXNlcl9uYW1lIjoiZ2VsYWRtIiwiZW1haWwiOiJnZWxhZG1AZ2VsY29ycC5jb20iLCJhdXRoX3RpbWUiOjE2NDQyMjM2MjksInJldl9zaWciOiJkYzNhNWJkMyIsImlhdCI6MTY0NDIyMzYyOSwiZXhwIjoxNjQ0MjU5NjI5LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0L1NBU0xvZ29uL29hdXRoL3Rva2VuIiwiemlkIjoidWFhIiwiYXVkIjpbInNjaW0iLCJjbGllbnRzIiwic2FzLioiLCJ1YWEiLCJvcGVuaWQiLCJzYXMuY2xpIl19.LKb29hMGUuubAamZuiHWCxBzeYrfH7gq5vRqwbTAqaS2uf-_QF-T-WfB15ovgjTWPjR69OuDPv3vlPaSc-9JGggLL79wong0jUGu8x2wI8sj2d3Sv3P7260SmvbSQOMtfrYqqkmuLgSnyOI5NKuNBUoY7RS_9Hkh2EjbbJ0k9pY8D1DPPytJ5WqtKnbReQppJgvby-v-yViHzcIfcth31uxSuJG8CD9UpnbWnF5SJMAr4-5YkMtqX1lZddLUp9ZrEpKmzuIjy6m9J44KNVts4SZq3NCCc4WeX0Q6N9l6c3GbGmhAH1t1LBFyItHGbAysf04_LY5xft7r5_P0vHN91Q\"},{\"name\":\"SASCLOUDNATIVE\",\"value\":\"1\"},{\"name\":\"SAS_LAUNCHER_DEPLOYMENT_ID\",\"value\":\"viya\"},{\"name\":\"SAS_MULTI_TENANCY_ENABLED\",\"value\":\"false\"},{\"name\":\"SASREDIRCMDOPTS\",\"value\":\"-addshell -noauth\"},{\"name\":\"SAS_CADENCE_NAME\",\"value\":\"lts\"},{\"name\":\"SAS_SERVICES_TOKEN\",\"value\":\"eyJhbGciOiJSUzI1NiIsImprdSI6Imh0dHBzOi8vbG9jYWxob3N0L1NBU0xvZ29uL3Rva2VuX2tleXMiLCJraWQiOiJsZWdhY3ktdG9rZW4ta2V5IiwidHlwIjoiSldUIn0.eyJqdGkiOiI1YTA1Mzc1MmVjMTQ0MTczYjU0NWU5MmE0YTZkNjE5ZiIsImV4dF9pZCI6InVpZD1nZWxhZG0sb3U9dXNlcnMsZGM9Z2VscmFjZSxkYz1jb20iLCJzdWIiOiI5OGEzZWE0ZS00YjUyLTQwZDEtYTRjYy0wMjhiYjQ1MjQyMmYiLCJzY29wZSI6WyJzYXN1c2VycyIsIkhSIiwiU2FsZXMiLCJwb3dlcnVzZXJzIiwic2FzYWRtaW5zIiwiY2xpZW50cy5yZWFkIiwiY2xpZW50cy5zZWNyZXQiLCJ1YWEucmVzb3VyY2UiLCJGaW5hbmNlIiwib3BlbmlkIiwidWFhLmFkbWluIiwiY2xpZW50cy5hZG1pbiIsInNjaW0ucmVhZCIsIlNBU0FkbWluaXN0cmF0b3JzIiwiY2xpZW50cy53cml0ZSIsIkdFTENvcnBTeXN0ZW1BZG1pbnMiLCJzY2ltLndyaXRlIl0sImNsaWVudF9pZCI6InNhcy5jbGkiLCJjaWQiOiJzYXMuY2xpIiwiYXpwIjoic2FzLmNsaSIsImdyYW50X3R5cGUiOiJwYXNzd29yZCIsInVzZXJfaWQiOiI5OGEzZWE0ZS00YjUyLTQwZDEtYTRjYy0wMjhiYjQ1MjQyMmYiLCJvcmlnaW4iOiJsZGFwIiwidXNlcl9uYW1lIjoiZ2VsYWRtIiwiZW1haWwiOiJnZWxhZG1AZ2VsY29ycC5jb20iLCJhdXRoX3RpbWUiOjE2NDQyMjM2MjksInJldl9zaWciOiJkYzNhNWJkMyIsImlhdCI6MTY0NDIyMzYyOSwiZXhwIjoxNjQ0MjU5NjI5LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0L1NBU0xvZ29uL29hdXRoL3Rva2VuIiwiemlkIjoidWFhIiwiYXVkIjpbInNjaW0iLCJjbGllbnRzIiwic2FzLioiLCJ1YWEiLCJvcGVuaWQiLCJzYXMuY2xpIl19.LKb29hMGUuubAamZuiHWCxBzeYrfH7gq5vRqwbTAqaS2uf-_QF-T-WfB15ovgjTWPjR69OuDPv3vlPaSc-9JGggLL79wong0jUGu8x2wI8sj2d3Sv3P7260SmvbSQOMtfrYqqkmuLgSnyOI5NKuNBUoY7RS_9Hkh2EjbbJ0k9pY8D1DPPytJ5WqtKnbReQppJgvby-v-yViHzcIfcth31uxSuJG8CD9UpnbWnF5SJMAr4-5YkMtqX1lZddLUp9ZrEpKmzuIjy6m9J44KNVts4SZq3NCCc4WeX0Q6N9l6c3GbGmhAH1t1LBFyItHGbAysf04_LY5xft7r5_P0vHN91Q\"},{\"name\":\"SASODBC\",\"value\":\"/opt/sas/viya/home/lib64/accessclients/lib\"},{\"name\":\"SAS_DEPLOYMENT_CONFIG_INSTANCE_ID\",\"value\":\"default\"},{\"name\":\"SAS_TRUSTED_CA_CERTIFICATES_PEM_FILE\",\"value\":\"/security/trustedcerts.pem\"},{\"name\":\"SYB_SAS_BULK_NOTE\",\"value\":\"1\"},{\"name\":\"SSL_CERT_FILE\",\"value\":\"/security/trustedcerts.pem\"},{\"name\":\"SAS_CADENCE_VERSION\",\"value\":\"2021.2\"},{\"name\":\"SAS_SPRE_VAR_PATH_SPOOL\",\"value\":\"spool\"},{\"name\":\"SAS_URL_SERVICE_SCHEME\",\"value\":\"https\"},{\"name\":\"SAS_CADENCE_DISPLAY_SHORT_NAME\",\"value\":\"Long-Term Support\"},{\"name\":\"SAS_LAUNCHER_PATH_SEPARATOR\",\"value\":\"/\"},{\"name\":\"SAS_LAUNCHER_DEPLOYMENT_INSTALL_PATH\",\"value\":\"/opt/sas/viya/home\"},{\"name\":\"SAS_LAUNCHER_SCRIPT_EXTENSION\",\"value\":\"sh\"},{\"name\":\"SAS_SCRIPT_EXTENSION\",\"value\":\"sh\"},{\"name\":\"SAS_CONSUL_URL\",\"value\":\"https://sas-consul-server:8500\"},{\"name\":\"SAS_CERTIFICATE_CA_CERTIFICATE_FILE\",\"value\":\"/security/ca.crt\"},{\"name\":\"SAS_SPRE_VAR_PATH_TMP\",\"value\":\"tmp\"},{\"name\":\"SASREDIRSASOPTS\",\"value\":\"-noauth\"},{\"name\":\"SAS_CERTIFICATE_PRIVATE_KEY_FILE\",\"value\":\"/security/tls.key\"},{\"name\":\"ODBCINI\",\"value\":\"/opt/sas/viya/home/lib64/accessclients/odbc.ini\"},{\"name\":\"SAS_LAUNCHER_INIT_LOCALE_DEFAULT\",\"value\":\"en_US\"},{\"name\":\"SAS_SPRE_APP_CONNECT\",\"value\":\"connectserver\"},{\"name\":\"SAS_DEPLOYMENT_CONFIG_PATH\",\"value\":\"/opt/sas/viya/config\"},{\"name\":\"SAS_URL_SERVICE_TEMPLATE\",\"value\":\"https://@k8s.service.name@\"}],\"envFrom\":[{\"configMapRef\":{\"name\":\"sas-tls-config-f8ccd48c6m\"}},{\"configMapRef\":{\"name\":\"sas-access-config-2bc27276tk\"}},{\"configMapRef\":{\"name\":\"sas-deployment-metadata-8h2k647tf9\"}},{\"configMapRef\":{\"name\":\"sas-launcher-lockdown-config-g96dc2km8g\"}},{\"configMapRef\":{\"name\":\"sas-programming-environment-path-config-dt5gfd629h\"}},{\"configMapRef\":{\"name\":\"sas-shared-config-k454gfd685\"}}],\"image\":\"crcache-race-sas-cary.unx.sas.com/viya-4-x64_oci_linux_2-docker/sas-programming-environment:1.23.41-20220107.1641559060733\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"sas-programming-environment\",\"resources\":{\"limits\":{\"memory\":\"2Gi\",\"cpu\":\"2\"},\"requests\":{\"memory\":\"300M\",\"cpu\":\"50m\"}},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"readOnlyRootFilesystem\":true},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"volumeMounts\":[{\"mountPath\":\"/tmp\",\"name\":\"tmp\"},{\"mountPath\":\"/sasuser\",\"name\":\"sasuser\"},{\"mountPath\":\"/sashelp\",\"name\":\"sashelp\"},{\"mountPath\":\"/config\",\"name\":\"config\"},{\"mountPath\":\"/opt/sas/viya/config/var\",\"name\":\"viya\"},{\"mountPath\":\"/security\",\"name\":\"security\"},{\"mountPath\":\"/opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts\",\"name\":\"security\",\"subPath\":\"cacerts\"},{\"mountPath\":\"/opt/sas/viya/config/etc/SASSecurityCertificateFramework/private\",\"name\":\"security\",\"subPath\":\"private\"},{\"mountPath\":\"/opt/sas/viya/home/commonfiles\",\"name\":\"commonfilesvols\",\"readOnly\":true}]}],\"dnsPolicy\":\"ClusterFirst\",\"imagePullSecrets\":[{\"name\":\"sas-image-pull-secrets-bft7d9f557\"}],\"initContainers\":[{\"env\":[{\"name\":\"KUBE_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.name\"}}},{\"name\":\"SAS_CERTFRAME_TOKEN_DIR\",\"value\":\"/certframe-token\"},{\"name\":\"SAS_ADDITIONAL_CA_CERTIFICATES_DIR\",\"value\":\"/customer-provided-ca-certificates\"}],\"envFrom\":[{\"configMapRef\":{\"name\":\"sas-certframe-config-2ch97fd95b\"}},{\"configMapRef\":{\"name\":\"sas-certframe-ingress-certificate-config-cmm2t44t88\"}},{\"configMapRef\":{\"name\":\"sas-certframe-user-config-c4ch2c59m7\"}}],\"image\":\"crcache-race-sas-cary.unx.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.20.12-20220111.1641909413651\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"sas-certframe\",\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"500Mi\"},\"requests\":{\"cpu\":\"50m\",\"memory\":\"50Mi\"}},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"readOnlyRootFilesystem\":true},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"volumeMounts\":[{\"mountPath\":\"/certframe-token\",\"name\":\"certframe-token\"},{\"mountPath\":\"/security\",\"name\":\"security\"},{\"mountPath\":\"/customer-provided-ca-certificates\",\"name\":\"customer-provided-ca-certificates\"}]},{\"env\":[{\"name\":\"SAS_K8S_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.name\"}}},{\"name\":\"SAS_K8S_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"SSL_CERT_FILE\",\"value\":\"/security/trustedcerts.pem\"},{\"name\":\"SAS_LAUNCHER_INIT_ENCODING_DEFAULT\",\"value\":\"utf8\"},{\"name\":\"SAS_LAUNCHER_INIT_LOCALE_DEFAULT\",\"value\":\"en_US\"}],\"envFrom\":[{\"configMapRef\":{\"name\":\"sas-go-config-7fkgm49fg2\"}},{\"configMapRef\":{\"name\":\"sas-shared-config-k454gfd685\"}},{\"secretRef\":{\"name\":\"sas-consul-client\"}},{\"configMapRef\":{\"name\":\"sas-programming-environment-path-config-dt5gfd629h\"}},{\"configMapRef\":{\"name\":\"sas-deployment-metadata-8h2k647tf9\"}},{\"configMapRef\":{\"name\":\"sas-programming-environment-java-policy-config-858m9t4b4m\"}},{\"configMapRef\":{\"name\":\"sas-batch-server-config-gk95dmd2h6\"}}],\"image\":\"crcache-race-sas-cary.unx.sas.com/viya-4-x64_oci_linux_2-docker/sas-config-init:1.9.3-20211012.1634048206086\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"sas-config-init\",\"resources\":{},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"readOnlyRootFilesystem\":true,\"runAsGroup\":1001,\"runAsUser\":1001},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"volumeMounts\":[{\"mountPath\":\"/tmp\",\"name\":\"config-init-tmp\"},{\"mountPath\":\"/config\",\"name\":\"config\"},{\"mountPath\":\"/opt/sas/viya/config/var\",\"name\":\"viya\"},{\"mountPath\":\"/security\",\"name\":\"security\"}]}],\"restartPolicy\":\"Never\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{\"fsGroup\":2002,\"runAsGroup\":2002,\"runAsNonRoot\":true,\"runAsUser\":4000,\"supplementalGroups\":[2003,3001,3003,3004,3006,3007]},\"serviceAccount\":\"sas-programming-environment\",\"serviceAccountName\":\"sas-programming-environment\",\"shareProcessNamespace\":true,\"terminationGracePeriodSeconds\":30,\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"workload.sas.com/class\",\"operator\":\"Equal\",\"value\":\"compute\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"tmp\"},{\"emptyDir\":{},\"name\":\"config-init-tmp\"},{\"emptyDir\":{},\"name\":\"sasuser\"},{\"emptyDir\":{},\"name\":\"sashelp\"},{\"emptyDir\":{},\"name\":\"config\"},{\"emptyDir\":{},\"name\":\"viya\"},{\"name\":\"certframe-token\",\"secret\":{\"defaultMode\":420,\"secretName\":\"sas-certframe-token\"}},{\"emptyDir\":{},\"name\":\"security\"},{\"configMap\":{\"defaultMode\":420,\"name\":\"sas-customer-provided-ca-certificates-29kdmk686c\"},\"name\":\"customer-provided-ca-certificates\"},{\"name\":\"commonfilesvols\",\"persistentVolumeClaim\":{\"claimName\":\"sas-commonfiles\",\"readOnly\":true}}]}}}}",
"k8sNamespace": "gelcorp",
"k8sSecrets": [
"SAS_SERVICES_REFRESH_TOKEN",
"SAS_SERVICES_TOKEN",
"SAS_CLIENT_TOKEN"
],
"kvPairs": null,
"limits": [],
"name": "doWork2mins",
"project": "",
"queue": "priority",
"requiredResources": [],
"requiredTags": [],
"restartable": true,
"startAfter": "",
"type": "KUBERNETESJOB"
},
"version": 1
}
],
"version": 1
}
We can also list all queues and view queue details:
/opt/sas/viya/home/bin/sas-viya workload-orchestrator queues list
collecting output...
{
"queues": [
{
"configInfo": {
"isDefaultQueue": true,
"maxJobs": -1,
"maxJobsPerHost": -1,
"maxJobsPerUser": -1,
"priority": 10
},
"name": "default",
"processingInfo": {
"state": "OPEN-ACTIVE"
},
"tenant": "uaa",
"version": 1
},
{
"configInfo": {
"admins": [
"geladm"
],
"maxJobs": -1,
"maxJobsPerHost": -1,
"maxJobsPerUser": -1,
"priority": 100,
"users": [
"geladm",
"Henrik"
],
"willRestartJobs": true
},
"name": "priority",
"processingInfo": {
"state": "OPEN-ACTIVE"
},
"tenant": "uaa",
"version": 1
}
],
"version": 1
}
We can also close, open, inactivate and re-activate queues (and perform similar tasks on hosts):/opt/sas/viya/home/bin/sas-viya --output text workload-orchestrator queues close-inactivate --queue priority
queue: priority
closing inactive queue: priority
collecting output...
printing output:
Queue
There's also another way to see queues, using the config command:
/opt/sas/viya/home/bin/sas-viya --output text workload-orchestrator config get
grid configuration:
Version 1
PassPhraseFile /opt/sas/viya/config/var/run/sgmg/default/sgmg.pwd
LicenseFile /opt/sas/viya/config/var/run/sgmg/default/setinit.sas
GuiDir
GuiZipDir
SharedDir .
Port 8080
NoSSL false
NoAuth false
UserNamesNoCase false
HostInactiveSecs 60
HostUnknownSecs 900
JobPurgeSecs 3600
JobUpdateSecs 15
ManagerUpdateSecs 15
RoundRobinSecs 0
SchedulerUpdateSecs 15
ServiceUpdateSecs 15
ServiceWaitSecs 15
JobChangeCmd
MaxStdOutLines 50
ProxyUrl
ProxyUser
ProxyPwd
UserResources
CompareOrder []
Admins [SASAdministrators]
TenantAdmins []
HostTypes [map[description:SAS Workload Orchestrator Server Hosts on Kubernetes Nodes maxJobsAllowed:-1 name:default role:server version:1]]
Queues [map[configInfo:map[] isDefaultQueue:true maxJobs:-1 maxJobsPerHost:-1 maxJobsPerUser:-1 name:default priority:10 processingInfo:map[] tenant:uaa] map[admins:[geladm] configInfo:map[] maxJobs:-1 maxJobsPerHost:-1 maxJobsPerUser:-1 name:priority priority:100 processingInfo:map[] tenant:uaa users:[geladm Henrik] willRestartJobs:true]]
UserGroups [map[admins:[Ahmed] name:Managers tenant:uaa]]
HostGroups []
Services []
This method also provides a way to update the configuration. We can save the (JSON) payload to a text file, update it (e.g. define a new queue), and re-apply it to the cluster using the set command:
/opt/sas/viya/home/bin/sas-viya workload-orchestrator config set --file /tmp/confignew.json
setting grid configuration:
file: /tmp/confignew.json
This is great for version control and for CI/CD-ing the grid configuration.
Last but not least, there's also a logs command which allows for adjusting thresholds for Workload Management-related log appenders, and viewing log messages. As always, be careful of making things too verbose unless troubleshooting an issue. Try it and the other plugins mentioned here in your Workload Management environment and refer to the SAS Administration documentation for more information.
Thank you for reading.
Find more articles from SAS Global Enablement and Learning here.
