Flushing the log before copying it

[Dodecaphonic]

I'm administering a SAS 9.3 Server on Unix. We want to prevent users from using proc printo so as to have a complete picture of what is being run on the batch server. We still want our users to have their logs copied wherever they choose. However, we would prefer not using the ALTLOG option (we have our reasons).

One solution we though of is to include, at the very end of the scheduled programs, an x command to copy the log file indicated in getoption(log):

%let current_log=%sysfunc(getoption(log));
x "cp &current_log some/destination/abc.log"; 

This technically works, but the log is not complete. Is there a way to "flush" the log so that all what needs to be written to the log file is written when we issue the cp command? (I know it will still be incomplete, but it's not an issue -- not having the x cp command in the copied version of the log is not a necessity.)

Thanks

EDIT 

It appears that sleeping for a few seconds gives good results:

%let current_log=%sysfunc(getoption(log));
data _null_;
  x = sleep(5,1);
  call system("cp &current_log /some/destination/LOG-COPY.log");
run;

But is this solid enough? Even increasing the sleep time, can I ever be certain that the log is up-to-date?

[jimbarbour · Accepted solution]

What's your logparm setting? If it's "buffered," try changing it to "immediate." Immediate is not as efficient, but log statements should be written as they are generated rather than filling a buffer and then written.

Jim

[Dodecaphonic]

It is indeed "buffered". So your suggestion would work... We'll need to see by how much it affects performance. Our server is not exactly a Ferrari, so this will play a big role in deciding whether we use it. Are you by any chance aware of any documented analysis / benchmarks that would assess the overhead?

Thank you

[jimbarbour]

---

@Dodecaphonic wrote:

 Are you by any chance aware of any documented analysis / benchmarks that would assess the overhead?

I'm not sure how much benchmarks run on another system would tell you about the impact in your particular environment.  In practice, here, it hasn't made a material difference.

What you might do is run a SAS batch job multiple times, half of the time with

-logparm "rollover=session write=immediate"

And the other half with 

-logparm "rollover=session write=buffered"

After you complete maybe three runs of each type, compare the run times.  If it lengthens the run time by, say, 10%, then no way; it's not worth it.  But I think it's going to be less than 5%, maybe a couple of minutes extra in a 1 hour job.  Immaterial, really.

Regarding directives such as #Y-#m-#d_#H-#M-#s and the like, I've found those effective from the command line as well as via RSUBMIT (and SYSTASK too if I recall correctly), but I'm not sure how they'd work if you're using the management console.  Can't hurt to try.

If you can use macro code in whatever part of the process defines the dataset and file names, then of course you can do something like &Year.-&Mon.-&Day._&Hour.-&Min.-&Sec or whatever you like.

Jim

[Dodecaphonic]

Thanks for the tips. Yes, the #Y #m #d work well in Management Console; the only limitation is that you can't use them to define environment variables, which could be useful in some scenarios.

[jimbarbour]

OK, so I have to confess some (significant) ignorance here.  Does Mgmt Console underneath the covers format a SAS batch command?

For example, when I do an RSUBMIT, I first build a command string and then set the SASCMD option to the value of the command string.

			%LET		Cmd_String				=	!sascmdv -logparm 'write=immediate rollover=session' -memsize 8G 			
						-altlog &LogPath\sub_process_logs\&Pgm._%Get_Date(FORMAT=YYMMDDD10)_%Get_Time(FORMAT=Dot_Time8).log;
			OPTION		SASCMD					=	"&Cmd_String";

When I finally do the actual RSUBMIT, the string I've built is used to launch the new SAS batch job.  If you can create such a command string, environment variables can be created via the SET statement.  For example, the following, when included as part of the command string, sets an environment variable called "DSN" and assigns it a value of "AB_Potential_Charts_2020."

-set DSN AB_Potential_Charts_2020

One could just as easily include a macro value in the command string for, say, a run date or an as of date or the like.

Then, inside the SAS program that is launched, I do a SYSGET to obtain the value of the environment variable.

%LET	&Var		=	%SYSGET(&Var);

I basically have a macro whose argument is the name of the environment variable.  Using SYSGET, I create a macro variable with the same name as the environment variable and use the macro variable inside the program.  

Perhaps that could be helpful; just thought I'd mention it.  I've found it particularly handy.

Jim

[Patrick]

altlog feels like the solution but if there are reasons not to use it then...

For running in batch you could add your copy script via -termstmt. You could consider to call a .sh (or .bat) file there as a child process also setting options noxsync and noxwait - and then have the wait in your script. Not sure if such an approach would allow SAS to fully terminate and also ensure that you get the full SAS log including the last "The SAS System used..." log entry.

Depending what you actually need this copy of the log for you could also consider ARM https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/armref/titlepage.htm 

[Dodecaphonic]

The reason why altlog is not our first choice is that we want to minimize manual edits of jobs and flows in the scheduling plugin of the Management Console. The more we can achive with SAS code the better. 

However, I'm curious how you'd make this work using a script... A supposed "copy-log.sh" would need to be aware of the log's location and name. The name being built with time variables (FLOWNAME_JOBNAME_#Y_#m_#d_#H_#M_#s.log), maybe an environment variable could be used, but if I'm not mistaken, those persist only inside a job time frame and environment... 

Thx

[Patrick]

While -termstmt executes the SAS session is still available so you could call a SAS Autocall macro there which then internally calls a .sh passing in the path and name of the log file to copy. But having said that: Adding -termstmt to your batch command line would require a similar change than adding -altlog

I agree that manually changing the generated batch commands in the deployment objects is not the way to go. Question is if that would be required or if you could add the altlog option via a config change - especially if applicable for all batch jobs under a app server context. Your generated batch commands will likely use sasbatch.sh so it might be worth to dive into this sasbatch.sh to identify where it calls "something" that's o.k. for site specific modification. sasbatch_usermods.sh sounds very promising.

How/where exactly to change the config will also depend on how your logging is configured (using the logging facility or not). https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lepg/p119kau8rt2ebgn1bzaipafu6jp3.htm 

[Dodecaphonic]

Makes sense, we'll make some experiments bearing all that in mind. Thanks again!

[Tom]

I don't understand your requirement.  You want to make two copies of the SAS log? 

Any ad hoc method like your mentioned method is not going to work.  Besides the timing issue you are asking about there is also the fact that any given program could have sent part of the log to some other file by using PROC PRINTTO.

You say you are running batch jobs so why not just include -altlog option in the batch job?

Why is -altlog not available to you?

[Dodecaphonic]

The goal is in fact to prevent the use of proc printto while allowing users to get a copy of their logs. For the rest, I refer you to the other replies.

[Tom]

When you want to run a SAS program you just type this command at the command prompt:

sas myprog.sas

And it will find the file myprog.sas in the current directory and write a files myprog.log and myprog.lst in the current directory.

If you want to have extra control there are dozens (hundreds?) of command line options you can include.  You can use -log to tell it where to write the log.  You can use -altlog to tell it to write a separate copy of the log somewhere else.

If you have some requirements about where logs are written then just create your own command that adds those options to the command line it uses to invoke SAS.   For example you might create a program named runsas so your command line is just:

runsas myprog.sas

And the result is that the following command is run:

sas -altlog /somedir/myuser_date_time.myprog.log myprog.sas

If you do it right the user is free to add any other command line options and they are passed into the call to the actual SAS executable.  So you could add -rsasuser option etc to your call to runsas.

Please explain why these option don't work for what you are doing.