SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

Will Flow be double triggered by file arrival event?

Reply
Contributor
Posts: 21

Will Flow be double triggered by file arrival event?

Hi,

Let's say we schedule a flow with the file arrival event and the file is ../dir/*.csv.

If in a short time, e.g. 5 minutes, two csv files are uploaded to ../dir/, will the flow be triggered twice?

Thanks!

Respected Advisor
Posts: 4,173

Re: Will Flow be double triggered by file arrival event?

Posted in reply to ASASProgrammer

If the event is "file arrival" only then I would expect that the flow gets triggered for each and every single matching trigger file file.

If you don't want this to happen then you would have to define a more advanced expression.

Contributor
Posts: 21

Re: Will Flow be double triggered by file arrival event?

Hi Patrick,

Is there any documentation on developing the expression?

The contents shipped with SMC don't contain the details.

Thanks,

Shi

Super User
Posts: 5,431

Re: Will Flow be double triggered by file arrival event?

Posted in reply to ASASProgrammer

I don't think there is much documentation about this.

What Patrick means is that you need to evaluate how this trigger is supposed to work. Maybe you can use some other wild card matching than *?

What is your requirement?

Data never sleeps
Respected Advisor
Posts: 4,173

Re: Will Flow be double triggered by file arrival event?

Posted in reply to ASASProgrammer

Hi Shi

An easy way to go could be:

1. File trigger on "arrival" using your wildcard matching.

2. SAS job querying the directory with the trigger files and checking if there is more than one matching the file pattern having a creation time >= system time minus 5 minutes

     - Exactly one matching file: SAS job creates a new trigger file with a different naming pattern

     - More than one matching file: SAS job deletes any trigger files with the new naming pattern (if any left over from a previous run).

3. File trigger on "exist" looking for this new trigger file with the different naming pattern

4. Rest of flow.

LSF itself has a rich command syntax but only a small subset is implemented into SAS Metadata (eg. the file trigger "arrival" which then creates the corresponding LSF command). Instead of using a SAS program checking for trigger files as I've suggested you could also write a script with LSF commands - "bhist" for example lets you query for job flows executed.

If not licensed separately then I believe LSF comes as an OEM version and only allows you to schedule SAS programs. So eventually you need to write a SAS programs which then calls the LSF script in order to submit such commands.

About documentation:

Schedule Manager documentation is available from within SMC help. There is also some online doc available, eg. here http://support.sas.com/documentation/cdl/en/scheduleug/64110/PDF/default/scheduleug.pdf

I found this - a bit outdated - paper which I liked quite a bit: http://support.sas.com/resources/papers/SASScheduling.pdf

You can get LSF documetation directly from Platform Computing (it's now owned by IBM) or also from the SAS site: Scalability Community: Platform Suite for SAS

Ask a Question
Discussion stats
  • 4 replies
  • 1068 views
  • 0 likes
  • 3 in conversation