topic Re: Count and List of SAS STPs used in Administration and Deployment

Count and List of SAS STPs used

shoin — Wed, 11 May 2022 18:54:14 GMT

I am trying to find the number of STPSs run in a given time period, most imp info would be the STP name and count.

Most STPs are run via SAS AMO (Excel) in my environment. (SAS 9.4 TS1M6-Office Analytics)

I have looked in EVM extended kit, I did not luck out there, the two STP related reports

Stored Process Daily Job Runtimes ARM Log Analysis - Stored Process daily job runtimes
Stored Process Job Profile by Runtime ARM Log Analysis - Stored Process Server job profile by job runtimes

renders nothing useful.

I ran a log parser which gives me counts but not which STP.

Has someone done this before and could offer any insight?

TIA

I ran a log parser

Re: Count and List of SAS STPs used

gwootton — Wed, 11 May 2022 20:17:46 GMT

If you have the SAS Environment Manager Service Architecture Framework this creates a table ARTIFACT.ARTIFACTUSAGEDETAILS with an "artifactType" and "artifact" column that can be used to find stored process usage.

proc sql;
create table stp as
select * from artifact.artifactusagedetails where artifacttype="STOREDPROCESS";
quit;

proc freq data=stp;
table artifact;
run;

Re: Count and List of SAS STPs used

shoin — Wed, 11 May 2022 22:18:21 GMT

Knowledge is indeed power ... I will give it a try. Much thanks!

Re: Count and List of SAS STPs used

shoin — Thu, 12 May 2022 18:39:44 GMT

The report came up with 0 obs. I ran the validation and also ensured that the APD ETL was also enabled (logs are being generated) so now I have to chase why despite validation for the framework and APM ETL initialization implemented and running, no observations come up.

Re: Count and List of SAS STPs used

gwootton — Thu, 12 May 2022 18:44:49 GMT

The information is read from the StoredProcessServer/PerfLogs files. Are these stored processes being run on the stored process server?

$ pwd
/sas94/config/compute/Lev1/SASApp/StoredProcessServer/PerfLogs
$ grep STOREDPROCESS *
arm4_STPServer_2021-07-07_comp_24779.log:2021-07-07T09:12:11,757|Perf.ARM./Products/SAS_Environment_Manager/Nightly_Reports/Audit_Reports_(Log_Forensic)/User_Accounts_Added.APPL| | |G,1941282731.757100,2,2,STOREDPROCESS,
...

Re: Count and List of SAS STPs used

shoin — Thu, 12 May 2022 21:32:27 GMT

Found the following in the logDiscovery_20220511_174502.log
--- start log ---
2022-05-11 17:45:05,987 INFO [main] [configureAPM] - ****************************
2022-05-11 17:45:05,987 INFO [main] [configureAPM] - found: StoredProcessServer
/opt/sas/config/Lev1/SASApp/StoredProcessServer
2022-05-11 17:45:05,987 INFO [main] [configureAPM] - PERF: SASApp StoredProcessServer
2022-05-11 17:45:05,987 INFO [main] [configureAPM] - Creating /opt/sas/config/Lev1/SASApp/StoredProcessServer/PerfLogs
2022-05-11 17:45:05,987 INFO [main] [configureAPM] - /opt/sas/config/Lev1/SASApp/StoredProcessServer/PerfLogs already exists.
2022-05-11 17:45:05,992 INFO [main] [configureAPM] - No Update: EMI content already in /opt/sas/config/Lev1/SASApp/StoredProcessServer/sasv9_usermods.cfg
2022-05-11 17:45:05,993 INFO [main] [configureAPM] - ****************************
2022-05-11 17:45:05,993 INFO [main] [configureAPM] - found: WorkspaceServer
/opt/sas/config/Lev1/SASApp/WorkspaceServer
2022-05-11 17:45:05,994 INFO [main] [configureAPM] - PERF_WS: SASApp WorkspaceServer
2022-05-11 17:45:05,994 INFO [main] [configureAPM] - Creating /opt/sas/config/Lev1/SASApp/WorkspaceServer/PerfLogs
2022-05-11 17:45:05,994 INFO [main] [configureAPM] - /opt/sas/config/Lev1/SASApp/WorkspaceServer/PerfLogs already exists.
... more lines
2022-05-11 17:45:06,539 INFO [main] [configureAPM] - ****************************
2022-05-11 17:45:06,540 INFO [main] [configureAPM] - Found 1 'Server.DataStepBatch' objects in metadata
2022-05-11 17:45:06,548 INFO [main] [configureAPM] - No Update: property value 'BatchServer.DataStep.Property.LogExt.xmlKey.txt' is correct
2022-05-11 17:45:06,548 INFO [main] [configureAPM] - Existing value is '_'

2022-05-11 17:45:06,549 INFO [main] [configureAPM] - Service Architecture Client APM Configuration completed
2022-05-11 17:45:06,550 INFO [main] [logDiscovery] - Running discovery...
2022-05-11 17:45:06,554 INFO [main] [logDiscovery] - Configuration file already exists at /opt/sas/config/Lev1/Web/SASEnvironmentManager/emi-client/Conf/log_locations.json, overwriting...
2022-05-11 17:45:06,654 INFO [main] [logDiscovery] - Discovered 13 matching log directories
2022-05-11 17:45:06,654 INFO [main] [logDiscovery] - [log_locations.json] written to Conf
2022-05-11 17:45:06,654 INFO [main] [logDiscovery] - Service Architecture Log Discovery completed successfully
--- end log ---
The STPs are run via SAS AMO Excel and under executing end user's credential (WSS).

Re: Count and List of SAS STPs used

gwootton — Fri, 13 May 2022 14:30:30 GMT

If you are running the stored processes in a workspace server then they are not logged in PerfLogs, so we can't report on them this way.

The logDiscovery process is what ensures all the application servers have the APM logconfig in place and updates the log_locations.json file so the log collection process knows where to get the logs from.

Re: Count and List of SAS STPs used

shoin — Fri, 13 May 2022 15:59:07 GMT

Thank you Greg. Is there any other way, since they are triggered via AMO; Excel.?

S

Re: Count and List of SAS STPs used

gwootton — Fri, 13 May 2022 16:51:23 GMT

You could add a custom logging configuration to the Workspace Server to capture stored process actions in a separate log and then parse those files. Workspace Server trace logging might help narrow down what logger you want.

I turned on trace logging and ran a stored process in a workspace server from the stored process web application and found the App.SASXSTP logger includes a line that provides the program path at the debug level:
2022-05-13T12:43:55,629 DEBUG [00000015] :sasdemo - [00000001] STPXMVA Input arg: '_program' = /Products/SAS Environment Manager/Dynamic Reports/Metadata Inventory/Server Properties

So if you added to your active logconfig.xml file (probably logconfig.apm.xml since you have the framework enabled) an new appender and a logger definition pointing App.SASXSTP at debug to that new appender it would create log files with this info. You could then parse them using SAS manually or by way of an ETL. This would only work going forward though, you'd have no historical data.

Re: Count and List of SAS STPs used

shoin — Sat, 14 May 2022 00:10:21 GMT

Much thanks! I will accept this as a solution. I discovered the same for WSS log (trace enabled) that it was including the pgm name just as you pointed out. What I will do is use the log parse I previously used (only generated count by user) to add a capture of the said line, place all WSS logs in one main file and run that parser on it. Much appreciate the input. I will circle back if this worked.