BookmarkSubscribeRSS Feed
ASASProgrammer
Calcite | Level 5

Hi,

Let's say we schedule a flow with the file arrival event and the file is ../dir/*.csv.

If in a short time, e.g. 5 minutes, two csv files are uploaded to ../dir/, will the flow be triggered twice?

Thanks!

4 REPLIES 4
Patrick
Opal | Level 21

If the event is "file arrival" only then I would expect that the flow gets triggered for each and every single matching trigger file file.

If you don't want this to happen then you would have to define a more advanced expression.

ASASProgrammer
Calcite | Level 5

Hi Patrick,

Is there any documentation on developing the expression?

The contents shipped with SMC don't contain the details.

Thanks,

Shi

LinusH
Tourmaline | Level 20

I don't think there is much documentation about this.

What Patrick means is that you need to evaluate how this trigger is supposed to work. Maybe you can use some other wild card matching than *?

What is your requirement?

Data never sleeps
Patrick
Opal | Level 21

Hi Shi

An easy way to go could be:

1. File trigger on "arrival" using your wildcard matching.

2. SAS job querying the directory with the trigger files and checking if there is more than one matching the file pattern having a creation time >= system time minus 5 minutes

     - Exactly one matching file: SAS job creates a new trigger file with a different naming pattern

     - More than one matching file: SAS job deletes any trigger files with the new naming pattern (if any left over from a previous run).

3. File trigger on "exist" looking for this new trigger file with the different naming pattern

4. Rest of flow.

LSF itself has a rich command syntax but only a small subset is implemented into SAS Metadata (eg. the file trigger "arrival" which then creates the corresponding LSF command). Instead of using a SAS program checking for trigger files as I've suggested you could also write a script with LSF commands - "bhist" for example lets you query for job flows executed.

If not licensed separately then I believe LSF comes as an OEM version and only allows you to schedule SAS programs. So eventually you need to write a SAS programs which then calls the LSF script in order to submit such commands.

About documentation:

Schedule Manager documentation is available from within SMC help. There is also some online doc available, eg. here http://support.sas.com/documentation/cdl/en/scheduleug/64110/PDF/default/scheduleug.pdf

I found this - a bit outdated - paper which I liked quite a bit: http://support.sas.com/resources/papers/SASScheduling.pdf

You can get LSF documetation directly from Platform Computing (it's now owned by IBM) or also from the SAS site: Scalability Community: Platform Suite for SAS

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 4 replies
  • 2930 views
  • 0 likes
  • 3 in conversation