This will be the first in a series of posts about the new interfaces for administering a SAS Grid Manager 9.4 M6 environment. M6 introduced a number of drastic changes to SAS Grid Manager, several of which have been the subject of earlier articles written by my colleagues in GEL. In this post, we’ll take a look at some of the ways in which a Grid Administrator might perform typical grid administration tasks using one of the new tools on offer; the SAS Workload Orchestrator Administration Utility, which is run entirely from the command line.
Overview
First, a quick recap. SAS Grid Manager is the name of the new offering in M6, which does not include Platform Suite for SAS. It is the new default grid offering, and is developed entirely by SAS; we no longer use Platform Suite for SAS as a third party application for grid computing by default (although a Grid offering that uses Platform Suite for SAS, called SAS Grid Manager for Platform, is still available). Since Platform Suite is not part of the setup, traditional administration tools like the SAS Grid Manager Module for SAS Environment Manager, the SAS Grid Manager Agent Plug-in for SAS Environment Manager, the SAS Grid Manager Plug-in for SAS Management Console and the LSF Command Line Interface that were around prior to SAS 9.4 M6 have also been replaced in or removed from the SAS Grid Manager offering. For a more detailed introduction, please read this post by Edoardo Riva.
The SAS Workload Orchestrator Administration Utility resembles the LSF Command Line Interface in that it provides a facility to monitor and administer the grid nodes, jobs and queues directly from the command line. The syntax of the tool is similar to that of the SAS Viya Command Line Interface (CLI).
Running the Administration Utility
The tool is located in <SAS-Config>/Lev1/Applications/GridAdminUtility. The tool can be invoked from the command line by running the sas-grid-cli.sh with an argument to perform the desired function. Use the --help option to see a list of all options available on the command line. See below for some basic examples.
Queues
To see a list of queues configured in the environment, run:
sas-grid-cli.sh --mhost <master-host> --mport <port> --grid-user <username> --grid-password <password> show-queues
SAS® Workload Orchestrator Administration Utility, Version 9.4M6 (build date: Nov 7 2018 @ 21:11:55)
Copyright© 2018, SAS Institute Inc., Cary, NC, USA. All Rights Reserved
{
"version": 1,
"queues":
[
{
"version" : 1,
"name" : "default",
"description": "",
"configInfo" :
{
"isDefaultQueue" : true,
"willRestartJobs" : false,
"priority" : 10,
"maxJobs" : -1,
"maxJobsPerUser" : -1,
"maxJobsPerHost" : -1,
"users":
[
],
"admins":
[
],
"hostNames":
[
],
"hostTypes":
[
],
"requiredTags":
[
],
"consumedResources":
[
],
"requiredResources":
[
],
"limits":
[
],
"preempts":
[
]
},
"processingInfo":
{
"state" : "OPEN-ACTIVE",
"jobsPending" : 0,
"jobsRunning" : 0,
"jobsSuspended" : 0
}
},
{
"version" : 1,
"name" : "highpriority",
"description": "High-priority jobs",
"configInfo" :
{
"isDefaultQueue" : false,
"willRestartJobs" : false,
"priority" : 20,
"maxJobs" : -1,
"maxJobsPerUser" : -1,
"maxJobsPerHost" : -1,
"users":
[
],
"admins":
[
],
"hostNames":
[
],
"hostTypes":
[
],
"requiredTags":
[
],
"consumedResources":
[
],
"requiredResources":
[
],
"limits":
[
],
"preempts":
[
]
},
"processingInfo":
{
"state" : "OPEN-ACTIVE",
"jobsPending" : 0,
"jobsRunning" : 0,
"jobsSuspended" : 0
}
}
]
}
In this example, we can see the details of the two queues we have defined. Specific queues can be queried by adding an additional option to the end of command; --queue <queue-name>. Queues can also be opened, closed, activated and inactivated, but queues cannot be created or deleted using the command line.
Hosts
To see information about grid hosts, run:
sas-grid-cli.sh --mhost <master-host> --mport <port> --grid-user <username> --grid-password <password> show-hosts
SAS® Workload Orchestrator Administration Utility, Version 9.4M6 (build date: Nov 7 2018 @ 21:11:55)
Copyright© 2018, SAS Institute Inc., Cary, NC, USA. All Rights Reserved
{
"version": 1,
"hosts":
[
{
"version": 1,
"name": "sasserver01",
"fqdn": "sasserver01.race.sas.com",
"aliases":
[
],
"ipAddresses":
[
"10.96.1.94","10.96.1.94"
],
"configInfo" :
{
"hostType" : "Masters",
"maxJobsAllowed" : 4,
"tags":
[
],
"scheduleThresholds":
[
],
"suspendThresholds":
[
]
},
"staticInfo" :
{
"buildDate" : "2018-11-08T02:01:07Z",
"cpuRating" : 376.186580,
"cpuRatingAllocated" : 0.000000,
"cores" : 4.000000,
"coresAllocated" : 0.000000,
"memory" : 32013.085938,
"memoryAllocated" : 0.000000,
"swap" : 799.996094,
"swapAllocated" : 0.000000,
"temp" : 203738.980469,
"tempAllocated" : 0.000000,
"osName" : "Linux",
"osDesc" : "CentOS 7.5.1804",
"cpuVendor" : "Intel",
"cpuArch" : "x86_64"
},
"dynamicInfo" :
{
"lastUpdateTime" : "2019-03-06T04:50:14Z",
"state" : "OPEN-OK",
"hostClosed" : false,
"licenseValid" : true,
"licenseType" : "GRID",
"licenseExpireDate" : "2019-02-20",
"utilization" : 0.032404,
"runQueue15s" : 0.133333,
"runQueue1m" : 0.083333,
"runQueue15m" : 0.112222,
"diskIORate" : 0.000000,
"netIORate" : 0.000000,
"ioRate" : 0.066623,
"pgRate" : 0.000000,
"usedMemory" : 4343.054688,
"usedSwap" : 0.000000,
"usedTemp" : 16168.773438,
"jobsRunning" : 0,
"jobsSuspended" : 0
}
},
{
"version": 1,
"name": "sasserver03",
"fqdn": "sasserver03.race.sas.com",
"aliases":
[
],
"ipAddresses":
[
"10.96.6.31","10.96.6.31","::ffff:10.96.6.31"
],
"configInfo" :
{
"hostType" : "Masters",
"maxJobsAllowed" : 4,
"tags":
[
],
"scheduleThresholds":
[
],
"suspendThresholds":
[
]
},
"staticInfo" :
{
"buildDate" : "2018-11-08T02:01:07Z",
"cpuRating" : 323.549890,
"cpuRatingAllocated" : 0.000000,
"cores" : 4.000000,
"coresAllocated" : 0.000000,
"memory" : 32013.085938,
"memoryAllocated" : 0.000000,
"swap" : 799.996094,
"swapAllocated" : 0.000000,
"temp" : 203738.980469,
"tempAllocated" : 0.000000,
"osName" : "Linux",
"osDesc" : "CentOS 7.5.1804",
"cpuVendor" : "Intel",
"cpuArch" : "x86_64"
},
"dynamicInfo" :
{
"lastUpdateTime" : "2019-03-06T04:50:00Z",
"state" : "OPEN-OK",
"hostClosed" : false,
"licenseValid" : true,
"licenseType" : "GRID",
"licenseExpireDate" : "2019-02-20",
"utilization" : 0.000666,
"runQueue15s" : 0.000000,
"runQueue1m" : 0.000000,
"runQueue15m" : 0.004444,
"diskIORate" : 0.000000,
"netIORate" : 0.000000,
"ioRate" : 0.000000,
"pgRate" : 0.000000,
"usedMemory" : 1059.351563,
"usedSwap" : 0.000000,
"usedTemp" : 1699.800781,
"jobsRunning" : 0,
"jobsSuspended" : 0
}
},
{
"version": 1,
"name": "sasserver04",
"fqdn": "sasserver04.race.sas.com",
"aliases":
[
],
"ipAddresses":
[
"10.96.10.116","10.96.10.116","::ffff:10.96.10.116"
],
"configInfo" :
{
"hostType" : "Masters",
"maxJobsAllowed" : 4,
"tags":
[
],
"scheduleThresholds":
[
],
"suspendThresholds":
[
]
},
"staticInfo" :
{
"buildDate" : "2018-11-08T02:01:07Z",
"cpuRating" : 401.172030,
"cpuRatingAllocated" : 0.000000,
"cores" : 4.000000,
"coresAllocated" : 0.000000,
"memory" : 32013.085938,
"memoryAllocated" : 0.000000,
"swap" : 799.996094,
"swapAllocated" : 0.000000,
"temp" : 203738.980469,
"tempAllocated" : 0.000000,
"osName" : "Linux",
"osDesc" : "CentOS 7.5.1804",
"cpuVendor" : "Intel",
"cpuArch" : "x86_64"
},
"dynamicInfo" :
{
"lastUpdateTime" : "2019-03-06T04:50:14Z",
"state" : "OPEN-OK",
"hostClosed" : false,
"licenseValid" : true,
"licenseType" : "GRID",
"licenseExpireDate" : "2019-02-20",
"utilization" : 0.000999,
"runQueue15s" : 0.000000,
"runQueue1m" : 0.000000,
"runQueue15m" : 0.006667,
"diskIORate" : 0.000000,
"netIORate" : 0.000000,
"ioRate" : 0.000000,
"pgRate" : 0.000000,
"usedMemory" : 1057.878906,
"usedSwap" : 0.000000,
"usedTemp" : 1699.843750,
"jobsRunning" : 0,
"jobsSuspended" : 0
}
}
]
}
Note that resource information is displayed for all hosts. You can view host information for for a specific host by adding --host <hostname> to the end of the command.
In addition to viewing host information, administrators can open (using open-host --host <hostname>) and close (close-host --host <hostname>) any of the grid hosts.
Jobs
sas-grid-cli.sh --mhost <master-host> --mport <port> --grid-user <username> --grid-password <password> show-jobs
SAS® Workload Orchestrator Administration Utility, Version 9.4M6 (build date: Nov 7 2018 @ 21:11:55)
Copyright© 2018, SAS Institute Inc., Cary, NC, USA. All Rights Reserved
{
"version" : 1,
"jobs":
[
{ "version" : 1, "id" : 3, "request" : { "name" :"testprogram", "queue" :"default", "project" :"", "user" :"sasdemo", "startAfter" :"", "restartable":false, "cmd" :"\/opt\/sas\/sasconfig\/Lev1\/SASApp\/GridServer\/sasgrid", "args": [ "\"SASBATCHSASPGM:'testprogram.sas'\"","SASWANTJOBINFO:1","\"SASBATCHWRKDIR:'\/opt\/sas\/gridwork\/sasdemo\/SASGSUB-2019-03-05_23.54.42.959_testprogram'\""], "envVars": [ ], "hosts": [ ], "requiredTags": [ ], "consumedResources": [ ], "requiredResources": [ { "name" : "cores", "value" : 0.000000 }, { "name" : "memory", "value" : 0.000000 } ], "limits": [ ] }, "processingInfo": { "state" : "RUNNING", "queue" : "default", "userGroup" : "", "suspendFlags" : 0, "queuePriority" : 10, "submitTime" : "2019-03-06T04:54:42Z", "startTime" : "2019-03-06T04:54:42Z", "endTime" : "", "lastCheckTime" : "2019-03-06T04:54:42Z", "launchStatus" : 0, "processId" : 21338, "executionHost" : "sasserver01", "exitCode" : 0, "totalTimePending" : 0, "totalTimeRunning" : 0, "totalTimeSuspAdmin" : 0, "totalTimeSuspThresh" : 0, "totalTimeSuspPreempt" : 0, "hosts": [ ], "requiredTags": [ ], "consumedResources": [ ], "requiredResources": [ { "name" : "cores", "value" : 0.000000 }, { "name" : "memory", "value" : 0.000000 } ], "limits": [ ], "limitValues": [ { "name" : "maxMemory", "value" : 0.000000 }, { "name" : "maxCpuTime", "value" : 0.000000 }, { "name" : "maxClockTime", "value" : 0.000000 }, { "name" : "maxIoTotal", "value" : 0.000000 }, { "name" : "maxIoRate", "value" : 0.000000 }, { "name" : "maxPageRate", "value" : 0.000000 } ], "stdOutput": [ ] } }
]
}
In this example, we can see one job running, in the default queue, submitted by user 'sasdemo'. Default behaviour of show-jobs is to display information for all running jobs, but there are several criteria by which we can filter, including by job-id (--id <job-id>), user (--user <username>), queue (--queue <queue>), host (--host <host>), state (--state PENDING|RUNNING|SUSPENDED|FINISHED|ARCHIVED|ALL), and more; use the --help option to see all options.
Jobs can also be suspended, resumed and cancelled using their job-id by using either of the of suspend-job --job <job-id>, resume-job --job <job-id> and cancel-job --job <job-id> commands.
Other commands and options
Output can be directed to a log file with the --log <filename> option. Use --debug or --trace to change verbosity.
The gridinfo command can be used to view the grid version, the current master and the expiry date of the current license. Administrators can also run sas-grid-cli.sh update-license --file <license-file-path> to renew the license from the command line.
SASGSUB
SASGSUB is the command-line client utility for submitting jobs, commands and programs to the grid for processing, without requiring SAS to be installed on the machine on which it is executed (but note that if you are using the SAS Grid Manager for Platform, it needs the LSF client installed). This utility will be familiar to those who have used earlier versions of SAS Grid software, and it works essentially the same way that it did previously, with one exception; in the new version, jobs will be run as the user whose credentials are provided when submitting the job (directly or via an authinfo file). Previously, jobs ran as the SASGSUB process owner.
The documentation provides a comprehensive list of arguments as well as sample commands for SASGSUB.
Final Points
Note that in the examples above, the output is displayed in JSON format. A soon-to-be-released hot-fix changes the default output to tabular format (Update: hot fix now available; please see SAS Note 63787). The JSON output can still be displayed by specifying --output json in the command.
Configuration changes cannot be made interactively from the command line using this tool, but administrators can specify a grid configuration file to use. The Workload Orchestrator Web Interface can be used to create (as well as load and update) a grid configuration file. We'll look at this application more closely in a later post.
Note that some of the commands are restricted to administrators; while any authenticated user can view job, host or queue information, they cannot close/open hosts, cancel/resume/suspend jobs, activate/inactivate queues, load configuration files, update licenses or view/modify logging information. Users can authenticate by passing in credentials (as per the examples above), or by referencing an authinfo file. Kerberos authentication is also supported; use the --sspi option instead of specifying credentials.
For more information about the grid in SAS 9.4 M6, view the documentation.
Thanks for reading. Please leave a comment below to ask questions or share your own experiences.
