Rake is a custom step and macro to extract Viya logs
This post is for those who are collecting and investigating Viya4 logs. I have created a custom step for users of SAS Viya Monitoring for Kubernetes to easily extract logs from SAS Studio. The name of this custom step is Rake (aka Kumade). The name comes from the rake that collects fallen leaves. The Rake has the following characteristics. If you prefer Bash to SAS Studio, there is also a shell script version of Rake.
- Run a macro from SAS Studio to extract more than 10,000 logs from OpenSearch.
- If the output argument is omitted, logs are output as a data set in WORK.LOG.
- If the log extraction period argument is omitted, logs are extracted from 15 minutes ago to the present.
- After the logs are extracted into a data set, FREQ or SGPLOT aggregates the logs and outputs the results.
- Date and time can be specified in E8601DT32.3 format, now, now-3h, etc.
- Check for common error patterns, count the number of cases, and set a flag on the output.
- Encodes username and password and stores them in a configuration file.
Prerequisite
How to use
Open SAS Studio and place the Rake.step file in the SAS Contents folder. Open the OpenSearch Login section for the first time and enter your username and password. Click Run to execute the custom step. The first run with default settings extracts the last 15 minutes of logs and creates a configuration file in My Folder. The second and subsequent runs will access OpenSearch using the encoded username and password stored in the configuration file.
The custom steps and their built-in macros are implemented to work without parameters as much as possible. Of course, you can explicitly specify the time period for which the logs should be extracted. There are several options that can be found in the custom step UI. Seeing is believing.
Results
There are two types of information output in the SAS Studio results: lists with macro arguments and execution time information, and frequency totals and plots. An example screen is shown below. The purpose of the frequency summaries and plots is to provide indications of errors from the extracted logs by date/time, container, level and type of client, and so on. Detailed descriptions are omitted, but reference is made to this as a means of exploring the logs.
Output data
The screenshot below shows that WORK.LOG is filtered by multiple criteria; the filter function in SAS Studio makes it easy to search for a specific log. When a message stored in WORK.LOG contains certain characters, the Check variable is set to a number. This number indicates the pattern of the string.
WORK.LOG Columns
The only variables output to WORK.LOG are fields that are commonly referenced in OpenSearch; the definitions of the variables in WORK.LOG are shown below.
| Colum Name | Label | Type | Length | Format |
| timestamp | timestamp %sysfunc(getoption(tz)) | Numeric | 8 | E8601DT23.3 |
| hour | hour(timestamp) | Numeric | 8 | |
| minute | minute(timestamp) | Numeric | 8 | |
| second | second(timestamp) | Numeric | 8 | |
| check | The number of the matched pattern | Numeric | 8 | |
| utc | UTC timestamp | Character | 32 | |
| id | _id field | Character | 64 | |
| level | level | Character | 16 | |
| logsource | logsource | Character | 64 | |
| container | kube.container | Character | 64 | |
| pod | kube.pod | Character | 64 | |
| message | message | Character | 4096 | |
| username | kube.labels.launcher_sas_com/username | Character | 32 | |
| client | kube.labels.launcher_sas_com/requested-by-client | Character | 32 | |
| patterm | String retrieved from the message field | Character | 132 | |
| n | Sequential number indicating the order | Numeric | 8 |
TSV File
The contents of WORK.LOG are output to /tmp as a TSV file. The file is created with a unique name that includes a timestamp. The output folder can be changed in the Options tab.
Configuration file
When a macro or custom step is run with an OpenSearch username and password, the following configuration file is created in the SAS contents. The macro creates a new configuration file if one does not exist. This configuration file defines encoded username and password and several patterns to check for errors. Add patterns identifying errors as needed. It is necessary to know if the error is occurring infrequently or constantly. The configuration file is in JSON format and can be edited in an editor. The configuration file can only be saved to SAS content; it cannot be saved to a directory on the SAS server.
- /Users/&sysuserid/My Folder/rakeConfig.txt
Prepare for errors
If you are the system administrator, it is recommended that you take the following log and configuration information when errors occur If you can formulate the process of obtaining them and pass them on to technical support before they ask, you can reduce the amount of time you spend going back and forth in the mail.
- get-k8s-info.sh
- sas-viya configuration download
- Extract OpenSearch logs at the time the error occurred with Rake
Reason for development
It was because I was having trouble extracting the logs I needed from OpenSearch. I created a macro that does it easily and included it in a custom step following the advice of a colleague. I hope this helps with your investigation.
