SAS Logs from LSF Scheduled jobs

[DBailey]

I'm trying to map the log file that SAS thinks it's using to what LSF actually uses so that we can email the log if the job fails.  In testing, I see this from SAS's perspective:

19 %LET LOG=%SYSFUNC(GETOPTION(LOG));
20 %put log=&log;
log=/mkt/Logs/RMA/Z_Test_z_test_con_#Y.#m.#d_#H.#M.#s.log

from the actual log is /mkt/Logs/RMA/Z_Test_z_test_con_#Y.#m.#d_#H.#M.#s.log_2025.03.18_08.51.10.log

How can I find the actual log?

[quickbluefish]

Is this a SAS Grid setup?

[DBailey]

yes.

[quickbluefish]

Assuming these are batch jobs, can you tell us what SAS Grid commands are being used to submit the jobs?  

For example, if you submit a job called test.sas that is located in /some/directory like this:

sasgsub -GRIDWAITRESULTS -GRIDSUBMITPGM "test.sas"

...then, when the job completes, the log (test.log) should be returned to the original directory /some/directory (and will in addition be located in a uniquely-named subdirectory of /some/directory that gets created for each run).  This is what the -GRIDWAITRESULTS option does.  

So if you submit that way, then the most recent log should be:  /some/directory/test.log

I assume you are trying to email the log from the Linux system rather than from SAS, correct?

[quickbluefish]

Here's another thing you can try -- if these are being submitted as batch jobs (e.g., from a Putty interface or other terminal), then you can add this very useful option to the sasgsub command:

sasgsub -GRIDSASOPTS "(-sysparm $PWD)" 

...along with whatever other sasgsub commands 

Then, in the SAS program, instead of doing the %sysfunc(getoption(... thing that you're currently doing, you can instead get the *actual* path from which the program was submitted by referencing the macro variable SYSPARM:

%let LOG_PATH=&sysparm;

...

[DBailey]

We use the LSF Scheduler to submit jobs.  I tested a few ways...and the most recent is that (from management console) the "command to run" is 

/sasconfig/compute/Lev1/SASApp/BatchServer/sasbatch.sh -log /mkt/Logs/RMA/Z_Test_z_test_con.log -batch -noterminal -logparm "rollover=session" -sysin /mkt/BatchJobs/z_test_con.sas

what got actually created was 

/mkt/Logs/RMA/Z_Test_z_test_con.log_2025.03.18_12.16.34.log

for the test, we have a macro that is included in z_test_con.sas that checks the syscc value.  If it is >4, then it sends an email alerting of a failure (using sas email processing).  I need to attach the lob to that email...but can't work out the actual log file name.

[quickbluefish]

I'm not sure what that shell script is doing exactly, but you might try adding this:

-sysparm $PWD

...and then in the SAS program, get the path like this:

%let log_path=&sysparm;

...and I think the name of the log file itself could just be derived from the SAS macro variable &sysprocessname by replacing ".sas" with ".log"

[quickbluefish]

It would help to see the shell script itself, because it's possible that it allows an argument like -GRIDWAITRESULTS

[gwootton]

This isn't related to LSF, the log name is a function of the Batch Server configuration.

The SAS Batch Server's logconfig.xml file defines the file name as:

<param name="FileNamePattern" value="%S{App.Log}%d{yyyy.MM.dd_hh.mm.ss}.log"/>

The %S{App.Log} translates to the value of the -log option.

The default configuration of the Batch Server specifies the log location (<SASConfig>/Levn/SASApp/BatchServer/Logs), and it's "Advanced Properties" has a Rolling Log Options field set to "_" and Output file extension set to .lst.

The Rolling Log Option could also be set to "_#Y.#m.#d_#H.#M.#s.log" if no logconfig.xml is being used.

So, when you deploy a batch job, the command line built by this is:
<Configured command path for batch server> -log <Configured Log Directory>/<flow_name>_<job_name><rolling log option> -print <configured output directory>/<flow_name>_<job_name><output file extension> -batch -noterminal -logparm "rollover=session" -sysin <path to job code in deployment directory>

It is possible to modify the command string built by the deploy job process so this could have been changed.

If the logconfig.xml file is being used, the FileNamePattern in the logconfig then uses the value of the -log option, appending the date and time to the end.
If the logconfig.xml is not being used (no LOGCONFIGLOC set) then the Rolling Log Option of "_#Y.#m.#d_#H.#M.#s.log" would similarly append the date and time to the end of the log name.

[DBailey]

which seems to match our config

  <appender class="FileAppender" name="BatchFileAppender">
      <param name="Append" value="false"/>
      <param name="Unique" value="true"/>
      <param name="FileNamePattern" value="%S{App.Log}%d{yyyy.MM.dd_hh.mm.ss}.log"/>
      <layout>
         <param name="ConversionPattern" value="%m"/>
      </layout>
   </appender>

Is there a way to map it back so that the executing job knows the actual log name?

[Quentin]

I don't know a way for the executing batch job to know the name of its log file.  I'm sure that has been wanted by many people for many years, but I don't remember ever having it.  I've always had to deduce the name of the log file (from the program name etc.).

One (ugly) work around is for your job to use PROC PRINTTO to write the log a new file.  If your job is responsible for creating the log file (rather than using the log file created by the shell script), it will know the name of the log file.  Typically in my jobs I will start with a PRINTTO step that sends my log to a file I create, in the same directory that admins chose for the log files.  This way I can process the log at the end, without worrying about the log naming scheme used by the admins.  But as I say, it feels like a hack.  For example, the log file is not the *full* log file from the job, it's only the log file from my code.

[gwootton]

I don't see anything referencing the actual file name of the log in the environment variables, macro variables or options, you could try doing something like pulling the value of the LOG option and the macro variables for the date and time (SYSTIME, SYSDATE) to try to find the file.

[quickbluefish]

Does the shell script not allow a "-sysparm" argument?

[gwootton]

The sasbatch.sh script that is run would allow the addition of the sysparm option, but the code line built by the deploy job process does not add one. However if we just wanted the log path we could extract it from the LOG option. The issue is that we would not know the generated suffix (e.g. 2025.03.18_08.51.10.log).

[DBailey]

This is what I did (which works in testing...not sure about production yet).  Based on what sas thinks the log is, we looked at that location for the files matching that description and took the one with the most recent timestamp.

%macro rma_error_alert();

%put RMA_ERROR_ALERT;
%put SYSCC=&SYSCC;


%if &SYSCC Gt 4 %then %do;
	
	
	%PUT _ALL_;

	%LET tLOG=%SYSFUNC(GETOPTION(LOG));
	%LET tWork=%SYSFUNC(GETOPTION(Work));	
	%let ActLog=X;

	%put tlog=&tlog;
	%put twork=&twork;
	%put SYSCC=&SYSCC;
	
	filename oscmd pipe "ls -1tr &tLog.*.*|tail -1";

	data _null_;
		infile oscmd truncover;
		input filename $100.;
		call symput('ActLog',filename);
		run;


	%put ActLog=&ActLog;

	filename tmail email	
		from=('xxxxx')
		SENDER=('xxxxx')
		to=( 'xxxxxxx')
		subject=('Production Error')
		%if "&ActLog" ne "X" %then %do;
			attach = ("&ActLog" ct="text/plain")
			%end;
		;
	
	data _null_;
		file tmail;		
		current= "SYSCC=&SYSCC";
		LogPath = "Log Path: &tLog.";
		Workpath = "Work Path: &tWork.";
		SEText = "SysErrorText: &SYSERRORTEXT";
		put "SYSCC=&SYSCC.";
		put "Logpath=&tLOG.";
		put "Workpath=&tWork.";
		put "SysError: &SYSERRORTEXT.";
		put ;
		put ;
		put "This email address is not monitored for replies.";
		put ;
	run;
	
	%end;

%mend;