11-12-2011 05:28 AM
Is it possible to deploy SAS DI Jobs programatically and then add them to a predefined job flow? I want to try and set up an LSF Job Flow for adhoc jobs when our normal etl batches have finished and have them added to it without user intervention.
Cheers in advance
11-13-2011 02:18 PM
May not be the best option, however you could
Schedule a flow that runs a job that dynamically builds "includes". The job could look at a specific folder where all the adhocs are saved and run all of them, of course you'll need to move the adhoc after being run other wise it will be run the next day also....
However doing this you lose all the goods things about metadata ie impact analysis etc.
11-15-2011 05:10 AM
I was hoping to avoid that and have it visible via the Job Schedule plugin in SMC...
A colleague and I did a similar thing for doing scheduling with base sas files and EG Projects a while back. It was basically a DI job that picked up all the EGuide and .sas files from a folder available in the folders tab of EG and created a number of .sas files with includes to each of them in 4 pre defined job flows. It had some smarts to make sure we didn't start a cottage industry of regularly scheduled EGuide projects performing ETL tasks. Great because it works but it gives very little visibility of what is running etc. via standard monitoring. All the monitoring had to be done via manipulation of log files or pre-defined error and status handling the user had to add to code. Most EG users don't both with such rigor.
I guess this raises another question;
Can the deployment location for DI jobs be controlled via authorisations/security? E.G. User in group A can deploy to batch folder x while Users in group b can deploy to batch folder y? Hmm may have to try that out...
11-18-2011 07:48 AM
Since most of this is metadata, I think this is possible to program by using the metadata API.
But after adding jobs to a flow, the flow has to be rescheduled so that LSF will be aware of the new jobs. If that's possible programmatically, I don't know.
11-19-2011 07:48 AM
Issuing a command to BSUB should fix the issue with resubmitting it. I doubt the metadata definition for a submitted job would actually permeate though from issuing a bsub command. I think it would need to be a two stage process;
12-22-2011 04:02 PM
In 9.3, the answer to this question is yes, we have added a command line interface for deploying di jobs programmatically. You can reference the online help or di studio users guide to see the specifics, and there is a wrapper script that handles the details that you can call from a command line.
12-29-2011 04:48 AM
Else: Building on what Barry proposes you could create a flow which calls wrapper jobs (calls in parallel). The wrapper jobs (a macro call) would be like: first node calls first job in list of .sas files in deployment directory for adhoc jobs, second node call second job... and so on - and if there is no n-th job then the wrapper code does nothing.
May be something like: First node creates SAS table with adhoc jobs, node 2 to node n are wrapper nodes calling an adhoc job and moving the job to an archive if execution is successful.... Something along this line.