Overview of rake-cli rake(aka kumade) is a script for extracting logs from SAS Viya Monitoring for Kubernetes. The name of the script, rake, is named after a tool used to collect fallen leaves. You can change the name of the script to kumade, râteau, Rechen, rastrello, pá zi, or whatever you like to make it easier to remember. I created the script based on a custom step I created.   Purpose rake was created to simplify log extraction from SAS Viya Monitoring for Kubernetes. OpenSearch has a 10,000-record limit for log extraction, making it difficult to retrieve logs before and after an error in a single operation. `rake` overcomes this 10,000-record limit, extracting logs within a specified time range.   To assist SAS Viya system administrators in error investigation. To extract logs from OpenSearch within a specified range. To extract logs in a format importable into Excel. To easily identify common errors.   Features rake offers the following features:    Extracts more than 10,000 lines of logs from OpenSearch in JSON or CSV format from a Linux terminal. Automatically creates a configuration file template on the first run. Extracts logs from the last 5 minutes if no arguments are provided. Allows specifying date and time using formats like E8601DT32.3, `now`, `now-3h`, etc. Extracts only feeds frequently referenced during investigations.  Outputs logs in JSON or CSV format. Adds fields to the output file including day, time, minute, second, and checks for common error patterns.   Introduction This section describes the prerequisites, placement, and configuration file for the script.   Prerequisites This section lists the required environment and configuration information to run the script. Environment SAS Viya SAS Viya Monitoring for Kubernetes Commands bash jq curl Configuration OpenSearch URL and api path OpenSearch username and password   Deployment Place the rake script in one of the following paths and set execution permissions using `chmod +x`:   $HOME/bin/rake /usr/local/bin/rake   Configuration File Creation and Editing After placing the script, run `rake` without arguments to create the configuration file. A template configuration file will be created if one doesn't exist.   $HOME/.rake/config.json Edit the configuration file to set the OpenSearch username, password, and OpenSearch endpoint. Modify the `api` value if a 404 Not Found error occurs during script execution.   endpoint api user passwd   After editing the configuration, running `rake` without arguments will extract logs from the last 5 minutes. If unable to connect to the server or if the username/password is incorrect, an error message will be displayed. Correct the values in the configuration file based on the error message. File permissions will be changed after each script execution to protect the password.   Error Patterns Common error patterns are registered in the `"default"."queries"` array of the configuration file. Pattern matching is performed in the order defined in the `queries` array. This feature facilitates easier error identification by matching extracted logs against these patterns. Users can add patterns by modifying the configuration file. Note that characters requiring escaping in JSON format cannot be used in patterns due to formatting constraints.   Configuration File Example   { "default": { "endpoint": "https://osd.example.com", "api": "api/console/proxy", "user": "admin", "passwd": "password", "queries": [ "The system has encountered an unhandled Exception", "ObjectOptimisticLockingFailureException", "ODBC SQL Server Wire Protocol driver", "Error creating compute session", "Error stopping CAS session", "Child terminated by signal", "ServerOperationException", "OAuth token is expired", "Unable to launch node", "JobExecutionException", "Internal Server Error", "Unhandled Exception", "SAS/TK is aborting", "Java heap space", "out of memory", "OutOfMemory", "Unexpected", "SSL error", "Exception", "Failure", "killed", "panic", "OOM" ] } }   Functionality This describes how to run `rake`, its arguments, and the output file format.   Example Execution rake is designed to run with default options. $ rake Processing... Extracted 4582 logs from the range now-5m to now. Saved the logs in the following file. -rw-rw-r-- 1 azureuser azureuser 2403970 Nov 15 00:19 /tmp/rake.20241115.001936.json   Example Options Examples are shown before the option descriptions.   rake Extracts logs from the last 5 minutes. rake -h  Displays help. rake -f now-10m  Extracts logs from the last 10 minutes. rake -f 2024-11-01T12:00:00 -t now-1h Extracts logs from the specified range. rake -c -o /tmp/foo.csv Saves logs to `/tmp/foo.csv` in CSV format. rake -f now-1w -m "SAS/TK is aborting" Extracts logs from the last week containing the specified string.    Options   -f _date/time_ Specifies the start point for log extraction. The default value is now-5m. Acceptable date/time formats are shown below: 2024-11-14T13:00:15  Local date/time 2024-11-14T04:00:15Z UTC date/time now Current time now-30s 30 seconds ago now-1m 1 minute ago now-1h 1 hour ago now-1d 1 day ago now-1w 1 week ago now-1M 1 month ago now-1y 1 year ago -t _date/time_ Specifies the end point for log extraction. The default value is now. Acceptable formats are the same as the -f option.   -o _output_ Specifies the output filename. The default filename is created under /tmp based on a timestamp.   -c Changes the output file format to CSV. The default output format is JSON.   -m _text_ Specifies the text to be included in the .message field as a log extraction condition. This option is useful for capturing specific error messages over a relatively long period. Special characters requiring escaping in JSON format (double quotes, {}, [], |) cannot be specified.   -e _endpoint_ Specifies a value to override the endpoint in config.json.  -u _user_ Specifies a value to override the user in config.json.   -p _password_ Specifies a value to override the passwd in config.json.   -l _limit_ Specifies the maximum number of times to call the OpenSearch API. The default is 100.   -h Displays usage information.   Output File The output file content is sorted in ascending order by timestamp. The output file format is either JSON or CSV. Both formats can be imported using Excel's "Text or CSV from". The output file items are shown below:   _index _id _source.@timestamp Hour of @timestamp Minute of @timestamp Second of @timestamp  _source.logsource   _source.level  _source.kube.pod  _source.kube.container _source.message  _source.properties.username _source.kube.labels.launcher_sas_com/username _source.kube.labels.launcher_sas_com/requested-by-client Number of matching patterns in message Pattern from config file matched with message sort Development Notes Developer notes are documented here.   Test Environment - SAS Viya LTS 2024.03 - SAS Viya Monitoring for Kubernetes 1.2.26 - OpenSearch 2.12.0   Troubleshooting   404 Not Found If the OpenSearch hostname is correct but the request path doesn't exist, modify the `api` value in `config.json`. To find the correct values, use OpenSearch Dev Tools and your browser's developer tools (F12) together. Open Dev Tools and display the browser's developer tools. Run the following query from the Dev Tools Console: GET viya_logs-*/_search { "query": { "match_all": {} } } Select the request from the Network tab of the developer tools. The `Headers/Request URL:` will show the endpoint and `api` values to specify in `config.json`. Depending on the version of OpenSearch, the Copy as cURL feature may be available at times. In some environments, the protocol was HTTP instead of HTTPS, and the API path was sometimes dashboards/api/console/proxy instead of api/console/proxy.   Testing Normal and error test patterns are stored in test/test.sh. When developers modify the script, they should run the test patterns to check that it is working correctly. ... View more

Rake is a macro and Custom Step to easily extract OpenSearch logs. This post is for those who are collecting and investigating Viya4 logs. I have created a custom step for users of SAS Viya Monitoring for Kubernetes to easily extract logs from SAS Studio. The name of this custom step is Rake (aka Kumade). The name comes from the rake that collects fallen leaves. The Rake has the following characteristics. If you prefer Bash to SAS Studio, there is also a shell script version of Rake.   Run a macro from SAS Studio to extract more than 10,000 logs from OpenSearch. If the output argument is omitted, logs are output as a data set in WORK.LOG. If the log extraction period argument is omitted, logs are extracted from 15 minutes ago to the present. After the logs are extracted into a data set, FREQ or SGPLOT aggregates the logs and outputs the results. Date and time can be specified in E8601DT32.3 format, now, now-3h, etc. Check for common error patterns, count the number of cases, and set a flag on the output. Encodes username and password and stores them in a configuration file.   Prerequisite   The custom steps have been tested on LTS 2023.03 and LTS 2024.03. If you use it with other versions, please test it. SAS Viya Monitoring for Kubernetes must be installed in order to run. An OpenSearch user and password are required.   How to use   Open SAS Studio and place the Rake.step file in the SAS Contents folder. Open the OpenSearch Login section for the first time and enter your username and password. Click Run to execute the custom step. The first run with default settings extracts the last 15 minutes of logs and creates a configuration file in My Folder. The second and subsequent runs will access OpenSearch using the encoded username and password stored in the configuration file.       The custom steps and their built-in macros are implemented to work without parameters as much as possible. Of course, you can explicitly specify the time period for which the logs should be extracted. There are several options that can be found in the custom step UI. Seeing is believing.   Results   There are two types of information output in the SAS Studio results: lists with macro arguments and execution time information, and frequency totals and plots. An example screen is shown below. The purpose of the frequency summaries and plots is to provide indications of errors from the extracted logs by date/time, container, level and type of client, and so on. Detailed descriptions are omitted, but reference is made to this as a means of exploring the logs.     Output data   The screenshot below shows that WORK.LOG is filtered by multiple criteria; the filter function in SAS Studio makes it easy to search for a specific log. When a message stored in WORK.LOG contains certain characters, the Check variable is set to a number. This number indicates the pattern of the string.     WORK.LOG Columns The only variables output to WORK.LOG are fields that are commonly referenced in OpenSearch; the definitions of the variables in WORK.LOG are shown below. Colum Name Label Type Length Format timestamp timestamp %sysfunc(getoption(tz)) Numeric 8 E8601DT23.3 hour hour(timestamp) Numeric 8   minute minute(timestamp) Numeric 8   second second(timestamp) Numeric 8   check The number of the matched pattern Numeric 8   utc UTC timestamp Character 32   id _id field Character 64   level level Character 16   logsource logsource Character 64   container kube.container Character 64   pod kube.pod Character 64   message message Character 4096   username kube.labels.launcher_sas_com/username Character 32   client kube.labels.launcher_sas_com/requested-by-client Character 32   patterm String retrieved from the message field Character 132   n Sequential number indicating the order Numeric 8     TSV File   The contents of WORK.LOG are output to /tmp as a TSV file. The file is created with a unique name that includes a timestamp. The output folder can be changed in the Options tab.   Configuration file   When a macro or custom step is run with an OpenSearch username and password, the following configuration file is created in the SAS contents. The macro creates a new configuration file if one does not exist. This configuration file defines encoded username and password and several patterns to check for errors.  Add patterns identifying errors as needed. It is necessary to know if the error is occurring infrequently or constantly. The configuration file is in JSON format and can be edited in an editor. The configuration file can only be saved to SAS content; it cannot be saved to a directory on the SAS server.   /Users/&sysuserid/My Folder/rakeConfig.txt   Prepare for errors   If you are the system administrator, it is recommended that you take the following log and configuration information when errors occur If you can formulate the process of obtaining them and pass them on to technical support before they ask, you can reduce the amount of time you spend going back and forth in the mail.   get-k8s-info.sh sas-viya configuration download Extract OpenSearch logs at the time the error occurred with Rake   Reason for development   It was because I was having trouble extracting the logs I needed from OpenSearch. I created a macro that does it easily and included it in a custom step following the advice of a colleague. I hope this helps with your investigation. ... View more

After reading the article "Promoting SAS Viya Job Flows", I realized that I am a lazy person. Creating a JSON file that defines what to export is very time-consuming and can keep me focused if I only have to do it once. But if I needed to export many jobs many times, I thought I would write a script. And I managed to create a working script.   The example screenshot shows the procedure of exporting a test flow "ant-flow". You can specify the name or ID of the flow as an argument to the script. Please install the sas-viya CLI before running this script, this will not work with Viya 3.x. The following shows how the script retrieves the desired ID and URI. First, get the ID of the flow from its name.   The next step is to use the ID of the flow as the key to search and get the URI of the flow and the jobs contained in the flow.   Retrieve the jobRequestUri by searching for Actions with the job ID as the key. The ID of the job flow is recursively broken down into jobs.   The script I attached is short and simple, and I think it can be easily modified. If you could point out any errors, I would be very grateful. ... View more

Inspired by the past post, I have created a script for starting/stopping SAS services in Windows using PowerShell.   The script identifies the SAS services and sorts them by configuration, level, and service start order to create the start and stop order. If you want to change the startup order, you can change the numbers in the script that indicate the sort order. I have made this script to work with as little configuration as possible. The script is simple enough that you can modify it to suit your particular environment. sas-servers.ps1   The script can be modified to add services to be captured and to modify the order of startup by modifying the following part. You can see how the script calculates the order of service startup by setting the $debug variable to $true and displaying the contents of the sort key.   You can download the script from the attachment. ... View more