12-02-2016 04:16 AM
I would ask you for an advice on how to better dial with the following problem...
We have a list of DI jobs that need to be schedule on the LSF, via the management console.
We crate the flow of jobs, but have this contraint.
One flow should start when a new file arrive on the folder.
If multiple file arrives, the flow must run for every file.
Each job in the flow must know the name (or some other information could also be good...) of the file that make the flow starts.
The problem is that we do not understand how to deal with this last step... Any advice?
12-02-2016 06:28 AM
12-02-2016 07:32 AM - edited 12-02-2016 07:35 AM
Dependencies/Sequencing Issues Between Files
You could create multiple clones of the jobs in DI if they are different files and if their is a dependency between the files. And then run them as seperate flows.
All Files Being Ran Through The Same Process
Otherwise you can stipulate the flow to start by saying "run when any of the conditions occur while scheduling in management console." This should pick up any files in the folder at that current time.
i.e. if you have multiple files that can trigger it will be set off by any one of the conditions being met.
Note: Would not hard code the file names in Jobs in DI, I would create a macro variable as the file name and loop through this as precode in the job to check if they exist. Then as Linas said loop the job in DI. This would be the most concrete solution.
12-02-2016 08:44 AM - edited 12-02-2016 09:20 AM
Hi all, and thanks for the suggestion. In our case, the cloning of the jobs is not an opinion, since the file that arriver are possibly infinite. The workflow is something like: user upload one or more files on the server, and the process start to validate and manipulate these files. Each file validation process is managed by a large set of jobs, that process each file for different steps of jobs. Eache set of job could fail, then the following steps should not be processed.
Things is a bit more complex, since many user can load different set of file, that should be processed by different flows. So, I'm wondering if there is any way, from SASMC, to pass parameters from a job to another, or maybe to make available to a job, or a list of jobs, of the file name that trigger the event that start the flow!
12-02-2016 09:28 AM
You could have an initial job which runs a System command to pick up the filenames and insert them into a sas array or sas variables. This would be essentially be polling any of the files. This job would then also generate the triggers required to run the specific flows.
Then dependant on the inputs, triggers for each of the flows are generated and using a system command it will output them into the required directory(s). And each flow could be triggered based on what's present in a folder at a given time.
That seems the most logical for what you require at the moment.