08-08-2013 04:41 PM
Let's say we schedule a flow with the file arrival event and the file is ../dir/*.csv.
If in a short time, e.g. 5 minutes, two csv files are uploaded to ../dir/, will the flow be triggered twice?
08-09-2013 07:44 AM
If the event is "file arrival" only then I would expect that the flow gets triggered for each and every single matching trigger file file.
If you don't want this to happen then you would have to define a more advanced expression.
08-09-2013 09:06 AM
Is there any documentation on developing the expression?
The contents shipped with SMC don't contain the details.
08-09-2013 09:26 AM
I don't think there is much documentation about this.
What Patrick means is that you need to evaluate how this trigger is supposed to work. Maybe you can use some other wild card matching than *?
What is your requirement?
08-09-2013 11:06 PM
An easy way to go could be:
1. File trigger on "arrival" using your wildcard matching.
2. SAS job querying the directory with the trigger files and checking if there is more than one matching the file pattern having a creation time >= system time minus 5 minutes
- Exactly one matching file: SAS job creates a new trigger file with a different naming pattern
- More than one matching file: SAS job deletes any trigger files with the new naming pattern (if any left over from a previous run).
3. File trigger on "exist" looking for this new trigger file with the different naming pattern
4. Rest of flow.
LSF itself has a rich command syntax but only a small subset is implemented into SAS Metadata (eg. the file trigger "arrival" which then creates the corresponding LSF command). Instead of using a SAS program checking for trigger files as I've suggested you could also write a script with LSF commands - "bhist" for example lets you query for job flows executed.
If not licensed separately then I believe LSF comes as an OEM version and only allows you to schedule SAS programs. So eventually you need to write a SAS programs which then calls the LSF script in order to submit such commands.
Schedule Manager documentation is available from within SMC help. There is also some online doc available, eg. here http://support.sas.com/documentation/cdl/en/scheduleug/64110/PDF/default/scheduleug.pdf
I found this - a bit outdated - paper which I liked quite a bit: http://support.sas.com/resources/papers/SASScheduling.pdf
You can get LSF documetation directly from Platform Computing (it's now owned by IBM) or also from the SAS site: Scalability Community: Platform Suite for SAS