topic Re: SASPy submit result of running .sas code in SAS Programming

SASPy submit result of running .sas code

Frank09 — Sun, 12 Jan 2025 09:08:11 GMT

Hi all,

I just wonder if I use SASpy library submit method to run .sas code in a SAS server, how should I know that the .sas code run successfully or not? Especially in automatic scheduling job using SASpy to run .sas code. Thanks!

Re: SASPy submit result of running .sas code

quickbluefish — Sun, 12 Jan 2025 18:00:40 GMT

Is the SAS Server running SAS Grid? Is this Linux or Windows?

Re: SASPy submit result of running .sas code

Ksharp — Mon, 13 Jan 2025 01:43:16 GMT

Check the LOG file generated by .sas file . In it ,if there are not any ERROR or WARNING message, then .sas is running successfully.

Check OS return code:

[sasuser2@sas1 scripts]$ ls -l
total 4
-rwxr-xr-x 1 sasuser2 sasusers 1125 Nov 28 17:10 remote_ifre.sh
[sasuser2@sas1 scripts]$ cat remote_ifre.sh
#!/bin/bash
SERVER1=10.7.70.39
SERVER1_PORT=22
SERVER1_USER=ifre1

SERVER2=10.7.70.40
SERVER2_PORT=22
SERVER2_USER=ifre1

#nc -z -w 2 ${SERVER1} ${SERVER1_PORT}
(sleep 1;)|telnet ${SERVER1} ${SERVER1_PORT} 2>&1|grep 'Escape character' > /dev/null 2>&1
if [ $? -eq 0 ];then
    echo "[+] ${SERVER1} detection successful."
    ssh ${SERVER1_USER}@${SERVER1} 'sh /data/ifre1/ifre.sh'
    if [ $? -eq 0 ];then
        echo "[+] Script exec success on ${SERVER1}."
    else
        echo "[+] Script exec failed on ${SERVER1}."
    fi

Re: SASPy submit result of running .sas code

sastpw — Mon, 13 Jan 2025 20:50:25 GMT

take a look at this part of the doc and see if it answers your question:
https://sassoftware.github.io/saspy/advanced-topics.html#automatic-checking-for-error-in-the-log-and-the-warnings-module

Re: SASPy submit result of running .sas code

sastpw — Mon, 13 Jan 2025 20:54:38 GMT

Also, you can access the SAS automatic macro variables as needed to check them:
https://sassoftware.github.io/saspy/api.html#saspy.sasbase.SASsession.SYSERR

as well as any macro variable using symget().

Re: SASPy submit result of running .sas code

Frank09 — Tue, 14 Jan 2025 00:40:18 GMT

Thanks! It is non-grid SAS server running on Windows platform.

Re: SASPy submit result of running .sas code

Frank09 — Tue, 14 Jan 2025 00:43:12 GMT

Thanks for your comments! I think it's better to search "ERROR" in latest SAS log file. However I don't know whether the WARNING message is really error.

Re: SASPy submit result of running .sas code

Ksharp — Tue, 14 Jan 2025 00:50:21 GMT

Yes. I think WARN msg is also another type of ERROR .
E.X. the LOG would display WARNING message: the length of variables from two datasets is different , that would lead to the value of variable be truncated .

Re: SASPy submit result of running .sas code

Frank09 — Tue, 14 Jan 2025 09:12:31 GMT

Thanks! This enhancement is convenient for us to catch ERROR through SASpy. Only check_error_log method is called to verified. User doesn't need to check log file himself. As the document points out, there is no perfect solution to know whether it's a real problem there. From automatic SAS job scheduling perspective, if we find there is ERROR in log file, no matter it's checked through check_error_log or ourselves, we can just report that scheduling job failed and let user to verify in further steps.

Re: SASPy submit result of running .sas code

Frank09 — Tue, 14 Jan 2025 09:20:18 GMT

Thanks! Use SYSERR macro is helpful if SAS or user defined procedures or steps write return code to syserr. From my understanding, it not as useful as checking ERROR in SAS log file.

Re: SASPy submit result of running .sas code

Frank09 — Tue, 14 Jan 2025 09:25:08 GMT

Thanks! I agree with you. If one WARNING message indicates a logic error problem, we can think it's an error. But if that WARNING is an error, why not report it as ERROR directly. There should be some cases of WARNING which are not errors. How do we distinguish them during automatic scheduling jobs?

Re: SASPy submit result of running .sas code

Ksharp — Wed, 15 Jan 2025 01:00:58 GMT

Nope. ERROR stands for your code have some error or is unable to execute it, but WARN is saying your code could perform sucessfully but could yield wrong result/output .
For example:
NOTE: An infinite likelihood is assumed in iteration 0 because of a nonpositive residual variance estimate.
WARNING: Stopped because of infinite likelihood.
NOTE: The above message was for the following BY group:
SampleID=1

Although your code could running but the model is not fittable or result is not trusted.
All you need to do is keeping LOG clearn to make result trust or correct.

Re: SASPy submit result of running .sas code

Frank09 — Thu, 16 Jan 2025 08:41:47 GMT

Thanks! I agree with you. For automatica SAS job scheduling, if there are ERRORs or WARNINGs in the sas log file, it should report the job is failed to notify user something wrong there.

Re: SASPy submit result of running .sas code

Tom — Thu, 16 Jan 2025 16:29:04 GMT

@sastpw wrote:

take a look at this part of the doc and see if it answers your question:
https://sassoftware.github.io/saspy/advanced-topics.html#automatic-checking-for-error-in-the-log-and-the-warnings-module

Do you know HOW the saspy code is doing this check? I have seen a lot of lazy log scanners that just search for the letters ERROR in the log and so flag CODE that is there to potentially write an error message when an issue is detected causing many false positives.

Also if you are searching for ERROR: as the note says then there are some error messages that SAS writes where there is other text between the end of ERROR and the colon character.

Re: SASPy submit result of running .sas code

sastpw — Thu, 16 Jan 2025 17:27:03 GMT

Yes, in fact, I do know HOW. I wrote SASPy 🙂

And, I've had to enhance how I was doing that a couple of times over the years to account for SAS idiosyncrasies, like having text between the ERROR and the :, and for cases where logging has been enhanced by the user such that the log contains other formatted content, like timestamps and pids and stuff either before or after the SAS log output - so ERROR isn't even in column 1 of the log I get back. Lots of crazy stuff.

Here's the check:

if re.search(r'\nERROR[ \d-]*:', logd):
warnings.warn("Noticed 'ERROR:' in LOG, you ought to take a look and see if there was a problem")
self._sb.check_error_log = True

Unfortunately, SAS was never completely consistent, like a regular programming language where you call a function and check a return code, regarding programmatically being able to asses the results from submitted code. As others have mentioned, sometimes you can see ERROR: in the log and there wasn't really an error that kept things from working. Sometimes something doesn't work and you don't get ERROR. WARNING can't preclude something not working, or prove it did. The system Macros like SYSERR SYSINFO and all the others, aren't 100% reliable. All that.

So, I try my best to provide these things so you can have the best chance of programmatically assessing your code. The best advice may be to only submit 1 SAS step of code at a time and use whichever method (Error check, Macro, ...) that works best for assessing that step, instead of submitting a whole program full of stuff and trying to assess all of it. Note you can also do things like, after a data step, check to see if the table exists and it has the number of rows expected.

SASPy is used in house for testing and that testing is automated and self assessing, using these various means to assess whether the code passed or failed. I wish there was simply one return code you could check for any SAS code you submit, but there isn't.

Tom

Re: SASPy submit result of running .sas code

Tom — Thu, 16 Jan 2025 18:35:46 GMT

So it looks like you are check for ERROR: starting in first column (or in your case immediately following linefeed) which means we don't need to write the kind of crazy code I have seen to avoid having ERROR: appear in the code that is echoed to the long.

Things like:

%put %str(ERR)OR: Macro parameter invalid.;
put 'ERR' 'OR: Value out of range.';

Re: SASPy submit result of running .sas code

sastpw — Thu, 16 Jan 2025 18:43:17 GMT

Not sure what that code is.

But, the example I showed was from one of the access methods that doesn't have the issue with user content showing up embedded prior to the actual SAS log line (I think it was IOM). The STDIO access method has to deal with that, and in it the check has to deal with that:

def _checkLogForError(self, log):
lines = re.split(r'[\n]\s*', log)
for line in lines:
if re.search(r'^ERROR[ \d-]*:', line[self._sb.logoffset:]):
return (True)
return (False)

Do you have some code where this isn't doing what you need?

Re: SASPy submit result of running .sas code

Tom — Thu, 16 Jan 2025 19:00:00 GMT

@sastpw wrote:

Not sure what that code is.

But, the example I showed was from one of the access methods that doesn't have the issue with user content showing up embedded prior to the actual SAS log line (I think it was IOM). The STDIO access method has to deal with that, and in it the check has to deal with that:

def _checkLogForError(self, log):
lines = re.split(r'[\n]\s*', log)
for line in lines:
if re.search(r'^ERROR[ \d-]*:', line[self._sb.logoffset:]):
return (True)
return (False)

Do you have some code where this isn't doing what you need?

Not talking about scanning the actual code. Just the SAS log itself. SAS logs echo the code, unless the user mistakenly set the NOSOURCE option.

Re: SASPy submit result of running .sas code

Tom — Fri, 17 Jan 2025 00:34:50 GMT

@sastpw wrote:

Not sure what that code is.

...

Do you have some code where this isn't doing what you need?

I don't really use SASPy but if you want to test if your logic makes the same mistakes as some of the other code scanners I was complaining about try running this program.

* Check if AGE is missing ;
data _null_;
  set sashelp.class;
  if missing(age) then put 'ERROR: Age is missing for ' name= ;
run;

You should get this as the LOG

1    * Check if AGE is missing ;
2    data _null_;
3      set sashelp.class;
4      if missing(age) then put 'ERROR: Age is missing for ' name= ;
5    run;

NOTE: There were 19 observations read from the data set SASHELP.CLASS.
NOTE: DATA statement used (Total process time):
      real time           0.10 seconds
      cpu time            0.06 seconds

And it should NOT be flagged as having an error.

Then try modifying the code so that it does print the error.

data _null_;
  set sashelp.class;
  if _n_=2 then call missing(age);
  if missing(age) then put 'ERROR: Age is missing for ' name= ;
run;

Display Manger will show the error message in RED.

And your error check code should also find that error message.

Re: SASPy submit result of running .sas code

Frank09 — Sun, 19 Jan 2025 22:29:47 GMT

Thanks for the comments! Excellent explanation! Would SASpy also provide another flag like?

check_warning_log

Thus, if either of check_warning_log or check_error_log is true, user can consider that the SAS program is not run smoothly.

From my perspective, we use Automation tool to schedule the .sas programs which are written by others. It's not easy to change the existing .sas program files to add additional checks on each data step.