working-dog Tracker

Re: how to copy viya 4.x studio .flw file from NFS to SAS content

working-dog — Fri, 28 Feb 2025 15:04:45 GMT

Not a timely response, but I too have looked into how to copy dataFlow and will comment on the results.

I was able to save the flow as a file using PROC HTTP. Here is a sample code of it.

/* Copy dataFlow to the directory */
%macro copyDataFlow(uri=, to=);
  %global rc;
  %local fref url n;
  
  %let rc=1;
  %let n=%sysfunc(monotonic());  
  %let fref=_LF%sysfunc(putn(&n, z5.));
  %let url=%sysfunc(getoption(SERVICESBASEURL))&uri;
    
  /* Assign file reference name. */
  filename &fref "&to" encoding='UTF-8';
  %if &sysfilrc^=0 & &sysfilrc^=1 %then %return;

  proc http method="GET" oauth_bearer=sas_services out=&fref url="&url";
  run;
  
  %if &SYS_PROCHTTP_STATUS_CODE=200 %then %do;
    %let rc=0;
  %end;
  
  filename &fref clear;
%mend;

%copyDataFlow(uri=/dataFlows/dataFlows/a96a3d9d-61cd-4f18-bdca-4149251f3d5a, to=/tmp/sample.flw);

Re: Rake is a custom step and macro to extract Viya logs

working-dog — Fri, 27 Dec 2024 06:28:18 GMT

Rake has been modified and a new ZIP file is attached.
The modification adds the ability to update configuration file credentials and OpenSearch URLs from the Custom Steps screen.

Rake is a shell script to extract Viya logs

working-dog — Mon, 18 Nov 2024 14:31:07 GMT

Overview of rake-cli

rake(aka kumade) is a script for extracting logs from SAS Viya Monitoring for Kubernetes. The name of the script, rake, is named after a tool used to collect fallen leaves. You can change the name of the script to kumade, râteau, Rechen, rastrello, pá zi, or whatever you like to make it easier to remember. I created the script based on a custom step I created.

Purpose

rake was created to simplify log extraction from SAS Viya Monitoring for Kubernetes. OpenSearch has a 10,000-record limit for log extraction, making it difficult to retrieve logs before and after an error in a single operation. `rake` overcomes this 10,000-record limit, extracting logs within a specified time range.

To assist SAS Viya system administrators in error investigation.
To extract logs from OpenSearch within a specified range.
To extract logs in a format importable into Excel.
To easily identify common errors.

Features

rake offers the following features:

Extracts more than 10,000 lines of logs from OpenSearch in JSON or CSV format from a Linux terminal.
Automatically creates a configuration file template on the first run.
Extracts logs from the last 5 minutes if no arguments are provided.
Allows specifying date and time using formats like E8601DT32.3, `now`, `now-3h`, etc.
Extracts only feeds frequently referenced during investigations.
Outputs logs in JSON or CSV format.
Adds fields to the output file including day, time, minute, second, and checks for common error patterns.

Introduction

This section describes the prerequisites, placement, and configuration file for the script.

Prerequisites

This section lists the required environment and configuration information to run the script.

Environment
- SAS Viya
- SAS Viya Monitoring for Kubernetes
Commands
- bash
- jq
- curl
Configuration
- OpenSearch URL and api path
- OpenSearch username and password

Deployment

Place the rake script in one of the following paths and set execution permissions using `chmod +x`:

$HOME/bin/rake
/usr/local/bin/rake

Configuration File Creation and Editing

After placing the script, run `rake` without arguments to create the configuration file. A template configuration file will be created if one doesn't exist.

$HOME/.rake/config.json

Edit the configuration file to set the OpenSearch username, password, and OpenSearch endpoint. Modify the `api` value if a 404 Not Found error occurs during script execution.

endpoint
api
user
passwd

After editing the configuration, running `rake` without arguments will extract logs from the last 5 minutes. If unable to connect to the server or if the username/password is incorrect, an error message will be displayed. Correct the values in the configuration file based on the error message. File permissions will be changed after each script execution to protect the password.

Error Patterns

Common error patterns are registered in the `"default"."queries"` array of the configuration file. Pattern matching is performed in the order defined in the `queries` array.

This feature facilitates easier error identification by matching extracted logs against these patterns. Users can add patterns by modifying the configuration file. Note that characters requiring escaping in JSON format cannot be used in patterns due to formatting constraints.

Configuration File Example

{ "default": { "endpoint": "https://osd.example.com", "api": "api/console/proxy", "user": "admin", "passwd": "password", "queries": [ "The system has encountered an unhandled Exception", "ObjectOptimisticLockingFailureException", "ODBC SQL Server Wire Protocol driver", "Error creating compute session", "Error stopping CAS session", "Child terminated by signal", "ServerOperationException", "OAuth token is expired", "Unable to launch node", "JobExecutionException", "Internal Server Error", "Unhandled Exception", "SAS/TK is aborting", "Java heap space", "out of memory", "OutOfMemory", "Unexpected", "SSL error", "Exception", "Failure", "killed", "panic", "OOM" ] } }

Functionality

This describes how to run `rake`, its arguments, and the output file format.

Example Execution

rake is designed to run with default options.

$ rake Processing... Extracted 4582 logs from the range now-5m to now. Saved the logs in the following file. -rw-rw-r-- 1 azureuser azureuser 2403970 Nov 15 00:19 /tmp/rake.20241115.001936.json

Example Options

Examples are shown before the option descriptions.

rake	Extracts logs from the last 5 minutes.
rake -h	Displays help.
rake -f now-10m	Extracts logs from the last 10 minutes.
rake -f 2024-11-01T12:00:00 -t now-1h	Extracts logs from the specified range.
rake -c -o /tmp/foo.csv	Saves logs to `/tmp/foo.csv` in CSV format.
rake -f now-1w -m "SAS/TK is aborting"	Extracts logs from the last week containing the specified string.

Options

-f _date/time_

Specifies the start point for log extraction. The default value is now-5m. Acceptable date/time formats are shown below:

2024-11-14T13:00:15	Local date/time
2024-11-14T04:00:15Z	UTC date/time
now	Current time
now-30s	30 seconds ago
now-1m	1 minute ago
now-1h	1 hour ago
now-1d	1 day ago
now-1w	1 week ago
now-1M	1 month ago
now-1y	1 year ago

-t _date/time_

Specifies the end point for log extraction. The default value is now. Acceptable formats are the same as the -f option.

-o _output_

Specifies the output filename. The default filename is created under /tmp based on a timestamp.

-c

Changes the output file format to CSV. The default output format is JSON.

-m _text_

Specifies the text to be included in the .message field as a log extraction condition. This option is useful for capturing specific error messages over a relatively long period. Special characters requiring escaping in JSON format (double quotes, {}, [], |) cannot be specified.

-e _endpoint_

Specifies a value to override the endpoint in config.json.

-u _user_

Specifies a value to override the user in config.json.

-p _password_

Specifies a value to override the passwd in config.json.

-l _limit_

Specifies the maximum number of times to call the OpenSearch API. The default is 100.

-h

Displays usage information.

Output File

The output file content is sorted in ascending order by timestamp.
The output file format is either JSON or CSV. Both formats can be imported using Excel's "Text or CSV from". The output file items are shown below:

_index
_id
_source.@timestamp
Hour of @timestamp
Minute of @timestamp
Second of @timestamp
_source.logsource
_source.level
_source.kube.pod
_source.kube.container
_source.message
_source.properties.username
_source.kube.labels.launcher_sas_com/username
_source.kube.labels.launcher_sas_com/requested-by-client
Number of matching patterns in message
Pattern from config file matched with message
sort

Development Notes

Developer notes are documented here.

Test Environment

- SAS Viya LTS 2024.03
- SAS Viya Monitoring for Kubernetes 1.2.26
- OpenSearch 2.12.0

Troubleshooting

404 Not Found

If the OpenSearch hostname is correct but the request path doesn't exist, modify the `api` value in `config.json`.
To find the correct values, use OpenSearch Dev Tools and your browser's developer tools (F12) together.
Open Dev Tools and display the browser's developer tools. Run the following query from the Dev Tools Console:

GET viya_logs-*/_search { "query": { "match_all": {} } }

Select the request from the Network tab of the developer tools. The `Headers/Request URL:` will show the endpoint and `api` values to specify in `config.json`. Depending on the version of OpenSearch, the Copy as cURL feature may be available at times.

In some environments, the protocol was HTTP instead of HTTPS, and the API path was sometimes dashboards/api/console/proxy instead of api/console/proxy.

Testing

Normal and error test patterns are stored in test/test.sh. When developers modify the script, they should run the test patterns to check that it is working correctly.

Rake is a custom step and macro to extract Viya logs

working-dog — Fri, 27 Dec 2024 06:58:15 GMT

Rake is a macro and Custom Step to easily extract OpenSearch logs.

This post is for those who are collecting and investigating Viya4 logs. I have created a custom step for users of SAS Viya Monitoring for Kubernetes to easily extract logs from SAS Studio. The name of this custom step is Rake (aka Kumade). The name comes from the rake that collects fallen leaves. The Rake has the following characteristics. If you prefer Bash to SAS Studio, there is also a shell script version of Rake.

Run a macro from SAS Studio to extract more than 10,000 logs from OpenSearch.
If the output argument is omitted, logs are output as a data set in WORK.LOG.
If the log extraction period argument is omitted, logs are extracted from 15 minutes ago to the present.
After the logs are extracted into a data set, FREQ or SGPLOT aggregates the logs and outputs the results.
Date and time can be specified in E8601DT32.3 format, now, now-3h, etc.
Check for common error patterns, count the number of cases, and set a flag on the output.
Encodes username and password and stores them in a configuration file.

Prerequisite

The custom steps have been tested on LTS 2023.03 and LTS 2024.03.
If you use it with other versions, please test it.
SAS Viya Monitoring for Kubernetes must be installed in order to run.
An OpenSearch user and password are required.

How to use

Open SAS Studio and place the Rake.step file in the SAS Contents folder. Open the OpenSearch Login section for the first time and enter your username and password. Click Run to execute the custom step. The first run with default settings extracts the last 15 minutes of logs and creates a configuration file in My Folder. The second and subsequent runs will access OpenSearch using the encoded username and password stored in the configuration file.

The custom steps and their built-in macros are implemented to work without parameters as much as possible. Of course, you can explicitly specify the time period for which the logs should be extracted. There are several options that can be found in the custom step UI. Seeing is believing.

Results

There are two types of information output in the SAS Studio results: lists with macro arguments and execution time information, and frequency totals and plots. An example screen is shown below. The purpose of the frequency summaries and plots is to provide indications of errors from the extracted logs by date/time, container, level and type of client, and so on. Detailed descriptions are omitted, but reference is made to this as a means of exploring the logs.

Output data

The screenshot below shows that WORK.LOG is filtered by multiple criteria; the filter function in SAS Studio makes it easy to search for a specific log. When a message stored in WORK.LOG contains certain characters, the Check variable is set to a number. This number indicates the pattern of the string.

WORK.LOG Columns

The only variables output to WORK.LOG are fields that are commonly referenced in OpenSearch; the definitions of the variables in WORK.LOG are shown below.

Colum Name	Label	Type	Length	Format
timestamp	timestamp %sysfunc(getoption(tz))	Numeric	8	E8601DT23.3
hour	hour(timestamp)	Numeric	8
minute	minute(timestamp)	Numeric	8
second	second(timestamp)	Numeric	8
check	The number of the matched pattern	Numeric	8
utc	UTC timestamp	Character	32
id	_id field	Character	64
level	level	Character	16
logsource	logsource	Character	64
container	kube.container	Character	64
pod	kube.pod	Character	64
message	message	Character	4096
username	kube.labels.launcher_sas_com/username	Character	32
client	kube.labels.launcher_sas_com/requested-by-client	Character	32
patterm	String retrieved from the message field	Character	132
n	Sequential number indicating the order	Numeric	8

TSV File

The contents of WORK.LOG are output to /tmp as a TSV file. The file is created with a unique name that includes a timestamp. The output folder can be changed in the Options tab.

Configuration file

When a macro or custom step is run with an OpenSearch username and password, the following configuration file is created in the SAS contents. The macro creates a new configuration file if one does not exist. This configuration file defines encoded username and password and several patterns to check for errors. Add patterns identifying errors as needed. It is necessary to know if the error is occurring infrequently or constantly. The configuration file is in JSON format and can be edited in an editor. The configuration file can only be saved to SAS content; it cannot be saved to a directory on the SAS server.

/Users/&sysuserid/My Folder/rakeConfig.txt

Prepare for errors

If you are the system administrator, it is recommended that you take the following log and configuration information when errors occur If you can formulate the process of obtaining them and pass them on to technical support before they ask, you can reduce the amount of time you spend going back and forth in the mail.

get-k8s-info.sh
sas-viya configuration download
Extract OpenSearch logs at the time the error occurred with Rake

Reason for development

It was because I was having trouble extracting the logs I needed from OpenSearch. I created a macro that does it easily and included it in a custom step following the advice of a colleague. I hope this helps with your investigation.

Re: ODSのSTYLEと画像の出力について

working-dog — Wed, 16 Aug 2023 03:31:58 GMT

実行環境によって使える設定場所が異なるので、当て推量込みで書きます。

SAS Enterprise Guideならば
- オプション>SASプログラム>サーバ接続時にSASコードをサブミットする。
- プロセスフローで使える Autoexec Process Flowを使う
SAS Studio
- オプション>Autoexec File
システム全体であれば
- !SASROOT配下の AUTOEXECに仕込む
  - WorkspaceServer配下の *_usermode.sas に仕込む

Script to make a list of export targets for Viya's job flow.

working-dog — Thu, 24 Feb 2022 11:05:57 GMT

After reading the article "Promoting SAS Viya Job Flows", I realized that I am a lazy person. Creating a JSON file that defines what to export is very time-consuming and can keep me focused if I only have to do it once. But if I needed to export many jobs many times, I thought I would write a script. And I managed to create a working script.

The example screenshot shows the procedure of exporting a test flow "ant-flow". You can specify the name or ID of the flow as an argument to the script. Please install the sas-viya CLI before running this script, this will not work with Viya 3.x.

The following shows how the script retrieves the desired ID and URI. First, get the ID of the flow from its name.

The next step is to use the ID of the flow as the key to search and get the URI of the flow and the jobs contained in the flow.

Retrieve the jobRequestUri by searching for Actions with the job ID as the key. The ID of the job flow is recursively broken down into jobs.

The script I attached is short and simple, and I think it can be easily modified. If you could point out any errors, I would be very grateful.

Start/Stop all SAS Services in Windows using PowerShell

working-dog — Mon, 06 Dec 2021 12:29:56 GMT

Inspired by the past post, I have created a script for starting/stopping SAS services in Windows using PowerShell.

The script identifies the SAS services and sorts them by configuration, level, and service start order to create the start and stop order. If you want to change the startup order, you can change the numbers in the script that indicate the sort order. I have made this script to work with as little configuration as possible. The script is simple enough that you can modify it to suit your particular environment.

sas-servers.ps1

The script can be modified to add services to be captured and to modify the order of startup by modifying the following part. You can see how the script calculates the order of service startup by setting the $debug variable to $true and displaying the contents of the sort key.

You can download the script from the attachment.

Re: SASとVBとの同期

working-dog — Mon, 18 Jun 2018 14:41:57 GMT

Async Propertyが使えるかも知れません。

手元に環境がないのですぐに確認できませんが、時間ある時に試してみます。

http://support.sas.com/rnd/itech/doc/dist-obj/comdoc/ilngca.html