Job Control and Scheduling

[Patrick]

Hi all

I have a job control table where I store at the beginning and the end things like jobname, job start and end time and also the job status/return code.

I've implemented this job control table update as a macro which I call at the very beginning and end of a job.

My question is:
What should I do that the macro call in the end executes whenever possible (also if there was a upstream error - i.e. because a source table didn't exist).

I'm aware that I best would implement such a task outside of SAS using some kind of wrapper script used to batch a SAS job and then collect return codes - but I would like a OS independent approach and can live with a 90% solution.

Any suggestions or links to papers very welcome.

What I'm also after is how to retrieve and store all this nice information like used CPU time and real time which SAS writes to the log in the end of a job. I know I've read somewhere about this and will find out - but if you happen to just know THE paper then please send me the link.


A second related area where I'm in the moment searching the right approach is:
I'm implementing ETL processes in a UNIX environment. Scheduling is right now solely time triggered using "naked" cron. All jobs are run sequentially. There is no other scheduler available.

I'm searching for an approach to run jobs in parallel together with job dependencies. I have an idea of how to do this using UNIX scripts which would allow me AND dependencies (cron starts the script, fork child processes, implement waits for child processes before batching dependent jobs).

If you know about some source of information dealing with scheduling in such an environment or if you have some real life experience then your feedback would be very welcome.


Thank in advance
Patrick

P.S: Environment is SAS 9.2 under UNIX

[Peter_C]

among the papers on the SAS website

Collecting Performance Statistics
http://support.sas.com/documentation/cdl/en/hosto390/61886/HTML/default/viewer.htm#/documentation/cd... unfortunately starts about stats on zOS, but much applies also to unix

more useful, might be
Sample 34301: Parse SAS Logs to Extract Performance and Timing Information
at http://support.sas.com/kb/34/301.html

peterC

[Patrick]

Thanks Peter

There is also a SUGI paper around the second link you've sent me.
http://www2.sas.com/proceedings/sugi30/219-30.pdf

Collecting performance statistics is for my purpose only a side story.

I should have mentioned that the main purpose of these control tables is to store the date of data loaded into target tables.
It's about maintaining history in tables so it's rather important that I keep data in sync and load it in the right sequence even if something falls over or a data delivery is delayed.

Cheers
Patrick

[Peter_C]

on zOS a couple of years ago I implemented some jcl that preceeded the standard batch invocation of SAS. The bit-on-the-front captured the equivalent of a process-ID (jobname and jes-jobID) with this info it created a tso command to capture a copy of the complete output of the SAS job currently running. The "bit-on-the-front" then launched another job which I refer to as a shadow job. The shadow job would only start once the original had finished;
1 shadow job would invoke tso in batch to invoke that command created earlier which filled a task-specific file with the original job's entire jes output (not only saslog but all other files written to "sysout" which included "job-messages" and "msgclass output" and any "ods listing" written to sysout).
2 it then invoked SAS again to parse that output file, collecting job messages as well as some application specific info from the SASlogs, and
3 in that final SAS step it would then arrange delivery to LAN areas (using the Connect Dierct service approved by the client for SSL delivery from zOS mainframe to servers outside of its security domain), delivering a copy of the SAS job-output files and
4 finally, SAS sent an email to named individuals (or team mailbox) with a links to the files written to the destination area of that SSL delivery.

That final SAS step was able to accomplish a lot of validation/monitoring of job processes (not only SAS).
The scheduling system to ensure the shadow job didn't start until the original had finished was simplistic on zOS (just use the same jobname again). I'm not sure if implementing a dependancy on the original job would be as easy with unix CRON or windows AT or your preferred job-scheduler.
The principle requirements were
1 that job dependancy
2 ability to collect the (required) SAS job outputs
3 extremely reliable (so no trying to be very clever 😉
4 that the original job would finish
We were fortunate to have a platform (zOS) which provided these requirements.

good luck
peter

[Patrick]

Hi Peter

If I would have JCL available for this implementation I wouldn't have started this thread... Unfortunately there is only CRON.

This nice z/OS behavior that only one job with the same job name can run at a time is also not what UNIX does - I can query if a specific file has been batched and is running before batching it again.

Both JCL and UNIX scripting is of course not OS independent. But I start to feel - and you confirm it in a way - that I should accept what I have and just make things work.

Thanks
Patrick

[data_null__]

> My question is:
> What should I do that the macro call in the end
> executes whenever possible (also if there was a
> upstream error - i.e. because a source table didn't
> exist).

I think there are OPTIONS to get SAS out of "error" mode OBS=0 SYNTAXCHECK so you could be sure the TERM macro executes. Might be worth looking into.

[Patrick]

Thanks data _null_

SYNTAXCHECK was the keyword to get me at the right starting point in the doc.

Cheers
Patrick