topic Will Flow be double triggered by file arrival event? in SAS Data Management

Will Flow be double triggered by file arrival event?

ASASProgrammer — Thu, 08 Aug 2013 20:41:30 GMT

Hi,

Let's say we schedule a flow with the file arrival event and the file is ../dir/*.csv.

If in a short time, e.g. 5 minutes, two csv files are uploaded to ../dir/, will the flow be triggered twice?

Thanks!

Re: Will Flow be double triggered by file arrival event?

Patrick — Fri, 09 Aug 2013 11:44:36 GMT

If the event is "file arrival" only then I would expect that the flow gets triggered for each and every single matching trigger file file.

If you don't want this to happen then you would have to define a more advanced expression.

Re: Will Flow be double triggered by file arrival event?

ASASProgrammer — Fri, 09 Aug 2013 13:06:41 GMT

Hi Patrick,

Is there any documentation on developing the expression?

The contents shipped with SMC don't contain the details.

Thanks,

Shi

Re: Will Flow be double triggered by file arrival event?

LinusH — Fri, 09 Aug 2013 13:26:18 GMT

I don't think there is much documentation about this.

What Patrick means is that you need to evaluate how this trigger is supposed to work. Maybe you can use some other wild card matching than *?

What is your requirement?

Re: Will Flow be double triggered by file arrival event?

Patrick — Sat, 10 Aug 2013 03:06:59 GMT

Hi Shi

An easy way to go could be:

1. File trigger on "arrival" using your wildcard matching.

2. SAS job querying the directory with the trigger files and checking if there is more than one matching the file pattern having a creation time >= system time minus 5 minutes

- Exactly one matching file: SAS job creates a new trigger file with a different naming pattern

- More than one matching file: SAS job deletes any trigger files with the new naming pattern (if any left over from a previous run).

3. File trigger on "exist" looking for this new trigger file with the different naming pattern

4. Rest of flow.

LSF itself has a rich command syntax but only a small subset is implemented into SAS Metadata (eg. the file trigger "arrival" which then creates the corresponding LSF command). Instead of using a SAS program checking for trigger files as I've suggested you could also write a script with LSF commands - "bhist" for example lets you query for job flows executed.

If not licensed separately then I believe LSF comes as an OEM version and only allows you to schedule SAS programs. So eventually you need to write a SAS programs which then calls the LSF script in order to submit such commands.

About documentation:

Schedule Manager documentation is available from within SMC help. There is also some online doc available, eg. here http://support.sas.com/documentation/cdl/en/scheduleug/64110/PDF/default/scheduleug.pdf

I found this - a bit outdated - paper which I liked quite a bit: http://support.sas.com/resources/papers/SASScheduling.pdf

You can get LSF documetation directly from Platform Computing (it's now owned by IBM) or also from the SAS site: Scalability Community: Platform Suite for SAS