topic Re: verify files are in folder- Dataflux in SAS Data Management

verify files are in folder- Dataflux

Rcflyboyjr — Tue, 18 Sep 2018 16:17:05 GMT

Using DataFlux. In standalone mode, meaning no other SAS environment is available.

I need to make sure there are files in a folder (name changes as the export date is appended to the filename)

before the Text File Import node is executed to prevent a "Error reading file" error.

My plan was to put an expression in the Process Flow checking to make sure at least one file exists.

then use If node to branch to the import process for that set of files .

I see in the expression user manual the fileexists function but nothing to allow passing dynamic names to it.

I thought about calling a batch file to return the DIR contents but cant seem to get the external node to work either.

Thank you

Re: verify files are in folder- Dataflux

VincentRejany — Thu, 20 Sep 2018 19:43:01 GMT

Hi

Are you able to know the file name or is there any green flag stating that the file is present and ready for being process?

Vincent

Re: verify files are in folder- Dataflux

Rcflyboyjr — Fri, 21 Sep 2018 15:32:32 GMT

The problem is the filenames are different each time we process.

I prefer the Text input node doesn't even get executed if there is no file in the folder .

Re: verify files are in folder- Dataflux

VincentRejany — Mon, 24 Sep 2018 09:18:18 GMT

Ok, it still possible to read the content of the folder before starting your process. What would be the rules for identifying the file to read?

Cheers

Re: verify files are in folder- Dataflux

Rcflyboyjr — Mon, 24 Sep 2018 14:55:03 GMT

Each week we get new files from one customer but once a month we get files from another customer. the files are currently in separate folders as I couldn't figure a way to only load cust_a files in TextNode A and cust_b in TextNode B pointing to same folder .. SO thus separate folders. Back to main question. on Week 1,2,3 there would only be files in one folder .. on week 4 we would have both sets of files. I preferred to have both processes started at once in one Process flow so the user only has one process to start.

When started it would

check that the input folder had files, if so branch to the actual import , cleanse and export .

if cust_b folder was empty it would branch to a complete section and wait for the other customer files to finish without erroring and stopping the process. BUT it should stop if there really was an error in one of the steps during load and cleanse.

I Think I might be able to do it with the other solution you provided on another question ( the listener demo )

Re: verify files are in folder- Dataflux

VincentRejany — Mon, 24 Sep 2018 16:30:56 GMT

Quite tough to provide you with the most consistent approach. However:

Path and file name can be built relying on macro variables

Therefore if there is one file in your folder, it is possible to read the folder, get the file name and pass it to the text input

If the format of the file does not change, you only have to pass the "PATH" and the text input node will read all the files in the folder applying the default structure.

You can also activate the FILENAME option for retrieving an additional column storing the name of the file.

I can provide you with the file watcher logic, but you usually need a "green ligh" file (of 0ko) which will tell you that you can process the main file. If your main file is pretty big and is uploaded by a ftp or another process, a file watcher logic may start to process it as soon as it is being created. This is why a green light flag needs to be created when the full upload is completed, and must be the one you are looking at for triggering the process.

Re: verify files are in folder- Dataflux

Rcflyboyjr — Wed, 26 Sep 2018 12:41:18 GMT

Vincent,

Thank you for the details. this sounds like what I have so far. I do use the macro variables. I split all the files into separate folders based on structure and I do use just the path. I also use the FILENAME feature to propagate the name through the process. I think based on your description and the similarities I am probably at the best solution for what I have. Since the files are large the header check process in the beginning IS slow as it has to read the entire file to strip out the header. I have requested python to be added to our environment so I think I will use that at the start of the process (external application node) to validate files exist for processing, verify header / column structure hasn't changed and then let the processes proceed.

Thank you again for taking the time to help and validate I'm on the right track.