11-19-2016 11:07 AM
On a Windows environment, I wrap my programs in two macros. The first directs the log and lst to permanent files via the PRINTTO procedure. The second closes the PRINTTO, parses the log for notes, warnings, and errors of interest, then sends the summary of the log check to the lst file.
This is not an option in batch submission, so I updated the first macro to write to the log via PRINTTO to an alternative location, namely "file.log" becomes file__BATCH.log". I then work on this file and delete it at the end after I have parsed its contents using the second macro and I keep the file specified by -LOG as the permanent log.
I am somewhat new to SAS batch. I have two questions:
1) Is my assessment correct that I cannot read from the file specified by the -LOG option in the current program, thus necessitating the second "shadow" log (I dislike the thought of possibility of writing *two* Gb sized logs, say if I were to be processing genomic data).
2) I invoke the SAS batch call using the X command. Is that an acceptable choice? I write each call using CALL EXECUTE.
11-19-2016 11:48 AM
You can read the log, immediatley after closing the log file by PROC PRINTTO; RUN;
as demostarted by next tested example:
proc printto log='/folders/myshortcuts/My_Folders/flat/test_log1.log'; data a; do x=1 to 5; y=x*3; output; end; run; proc printto; run; filename mylog '/folders/myshortcuts/My_Folders/flat/test_log1.log'; data _null_; infile mylog truncover; input log_line $100.; file print ods; put @3 '--- ' log_line $100.; run;
adapt it to your needs, either OnLine or Batch.
11-19-2016 04:49 PM
SAS batch programs can be more easily run from a scheduler. Where are you running these from - your PC or a SAS server? If you are using a SAS App server then SAS Management Console can be used for running batch programs via a scheduler, either now or at a particular time.
11-19-2016 08:13 PM
What's the purpose of parsing the logs?
If this is just for controling the job flow then using any of the available schedulers within SMC will allow you to define such flows which check for condition codes (=job ended with Warning or with Error) and you can then stop the flow or branch into some Error/Warning condition logic (=calling different jobs depending on return codes from one or multiple pre-decessors).
Defining flows via SMC makes also things like controlled parallel processing of jobs a rather simple task.
If you implement the log parsing bit within the same job which you want to control, then you'll never get to this code bit in case of a serious up-stream error condition in your code. It's also for this reason better to keep such logic outside of the job you want to control.
11-20-2016 02:22 AM
I'm running SAS on UNIX, so I have more and better tools at hand, but I do all my log-checking in the shell script that runs my batch jobs. I mainly use grep to scan for certain phrases that signal "problematic" runs, and set special return codes that the scheduler can react to.
11-20-2016 11:45 AM
I have not heard a convincing argument against using SAS to run SAS programs in batch. Whereas I appreciate reading the log with a script, it implies that that the run completed. I prefer using SAS to check the log, since I could easily modify the code to collect the results in a data set that could be used to report metrics.
11-20-2016 12:13 PM
There is nothing to prevent you from using SAS to check the log. What they are saying is that counting on the current SAS session to check its own log is asking for trouble. If the program fails then how can it reliably check its own results?
You can use the MP Connect features of SAS/Connect to run the pieces of code you want to be able to check in separate sessions.
Or if you don't have SAS/Connect licensed then just have one SAS program write our a program and run SAS via tools like SYSTASK or PIPEs.
11-20-2016 03:37 PM
In my experience even a program that fails can almost always check it's own results.
If the progam has entered syntax check mode (obs=0 etc), it can recover from that with:
options obs=max replace NoSyntaxCheck;
I think it's safe to end a program with a self-scanning %logcheck. My appraoch writes the results ot the logcheck to the log, so one feature is just the ability to add a log summary to the end of the log.
I suppose if SAS really crashes hard (like a corrupted SAS session) the above approach wouldn't recover. But that's pretty rare in my experience.
For my scheduled jobs, I have them scan their own log, and email me if there is a problem, and write a record to a JobLog datasets with the name of the program, logfile, and number of errors/warnings/bad notes found in log (hopefully 0). And I have a separate SAS job that runs daily and checks the JobLog dataset to see that all the of the expected jobs were run in the past 24 hours. If a job crashed hard enough that the log checker did not run, an expected record would be missing from the JobLog dataset.
The weakness in my approach is that all the checking is done by SAS jobs. So if the SAS server is down, or even if the LSF scheduler hangs for some reason, I have to notice I didn't get an "All clear" email.
11-28-2016 03:00 PM
Thank you for your reply. If the program fails, then the log check fails, as you state. Without a log check and other details, I consider the run a failure and would re-run it. My understanding is that the log would still persist, so I could read it using the log checking macro, but it only report the line with the message of interest. Often the issue sprawls across lines or the line is just the message without details, requiring that I open the log.
I arrived at using -ALTLOG to direct the permanent log. The programs all use the PRINTTO procedure, so I direct that log to xxx__batch.log when SYSIN is not missing. I read this log with the log checking program and delete it at the end. The results of the log checking log appears at the bottom of the lst output. To me, this has been reliable.
Interactively, I could close the log, run the log checker (accept that this action would be omitted from the log), and direct the results to the print = location. In batch, SAS keeps the log lock until the program terminates.
11-20-2016 03:23 PM
Hope all is well.
I do similar log scanning to what you describe, and I think it's a fine approach.
My memory is on linux, a batch program can in fact read its own current log, without a problem.
On Windows the last time I tried, that wouldn't work. So for batch jobs on windows my %logcheck macro would typically start by using systask to invoke windows command to copy the current log file to a temp file, then the temp file could be scanned. If the log file is several GB, I guess that could take a little time to copy, but still typically a small fraction of the SAS job time. I was surprised that even though I couldn't read the current log file, I was able to copy the current log file to a second file.
But I've moved toward using the PROC PRINTTO approach for the fast few years, because I'm working more on stored processes and similar server stuff, and I typically want the log files to be written somewhere that is useful to me, rather than the default location chosen by the server admin. So as you describe, I start code with a PROC PRINTTO log= ... ; and end code with a PROC PRINTTO log=log;run; and then scan the log file.
I'm not understanding why you say your current PROC PRINTTO approach is not an option for batch submissions, or how the approach you describe in the first paragraph is different than the appraoch you describe in the second paragraph.
1) Yes, I believe you are correct that in Windows you cannot read from the "current" log file (specified in -LOG for a batch job), because it is locked. But if you use PROC PRINTTO log=mylog.log approach, that willl not write to two log files that the same time. It redirects the log, so log lines are written to the new log file instead of the old one. So you're not writing two huge log files. I don't think.
2) Yes, I think X statement is probably fine. You might want to look into SYSTASK statement. It gives a bit more control with macro vars for return code, and waiting for tasks to complete.
11-28-2016 03:25 PM
I wonder if this is the Quentin of SAS-L fame?
I think you might find -ALTLOG a better approach than copying the log, unless you do it conditionally and the log is large (why always make 2 Gb+ logs?). In this scenario, you do not use -LOG, but rather have the program redirect a temporary log via the PRINTTO. It took me a while to starting using DM to capture all of the log before PRINTTO procedure, meaning I missed the header and the results of the autoexec.sas, which are informative when this handy option in the .cfg file: -ECHOAUTO. DM does not seem to work in batch, but please correct me if I am wrong.
To clarify about PRINTTO was not an option, I was attempting to close the current batch log before I read it. I will test this when I have more time, but we are on deadlines. I was receiving an ERROR when I tried to INFILE the log, since it was still locked.
11-28-2016 05:59 PM - edited 11-28-2016 06:02 PM
Just on a side note and so you don't have to replicate the full log: Are you aware of Proc SCAPROC? May be that could give you already all the log parsing you need.
sas yourjob.sas -initstmt "proc scaproc; record 'yourjob.txt' ; run;"
11-28-2016 09:32 PM
No, I was unaware of this procedure. When I get some breathing room, I will explore it. I spent the early 2000's being wrong on SAS-L, a traditional I seem to be proudly continuing in the late 2010's on SAS Communities Oh well, at least I learn
PS These logs are quite small.