Oddball Error with Copy Files Task - resolving macro expressions

[PhilGUK]

This is a blast from the past - this thread from 2015. Solved: Oddball Error with Copy Files Task - SAS Support Communities
I'm on SAS EG 8.2 Update 5 (8.2.5.1277) 

I've been converted a number of our projects (and creating from new) from using PC Files Server to using EG Copy File Task, coupled with proc import/export Unix to Unix and then  Copy Files task to return the .xlsx to Windows.  We have had a few password/access issues and a couple of other things that PCFS has evolved into doing that are clumsy, which I want to avoid. I also see Copy Files Task as a stepping stone away from PCFS or SAS Base (Windows) as our organisation moves to Viya on cloud.  I too love the flexibility Copy Files Task offers and usually set it up with macro vars for input_path, file_name and target_path, in a parameters file so I can comment the heck out of it.

I get the same problem as in 2015.  Sometimes (quite often) the code node following the CFT starts to run before the CFT has finished.  The job I normally run next is to delete the file from Unix which the CFT is moving.

It gives the same as in 2015: the log just shows the "resolving macro expressions" and does not complete the transfer.  

Unlike the 2015 post, I already have this a predecessor linked to the next code node, so the 2015 solution does not work for me.

I had thought of putting the next code node to sleep (data _null_; call sleep(15,1); run;) to try and hold it back.  Given @ChrisHemedinger  reply in 2015 about this being (possibly) related to timing and the CFT task using a background thread, which the subsequent job, finishing quicker then terminates the CFT, I am hopeful this may work. Nope, perhaps related more to the time it starts.  When I run it a second time, from the Export code node, it works fine.  In both cases, it looks like the Delete Excel Unix code is waiting (in yellow).  So this does seem to be a first time thing.  When I hit this on another project, I broke the flow after the CFT, but that is a bit clunky. This seems to me like it fails the first attempt, then I struggle to make it fail again.

I thought it was just me!  Anyone else having this same kind of issue and did you find a away past it?  If sleeping in the next step doesn't work (which it just didn't), then I feel my only option left is to remove the delete temp unix file bit from that flow (perhaps give it a flow all of it's own).  Or perhaps I should throw in a few code boundaries quit; run; at the end of the export.  Not found a way, yet...

[PhilGUK]

I was just helping someone with a Copy Files Task and noticed two things:
1. Unlike a code code, there is no option to select Server on a CFT

2. Whilst moving a 230Mb file (which for some reason took 9mins - our server must be busy, or bad home Wifi or VPN), we ran another code node.  I was expecting this to be queued, but it was not.

So this sort of confirms Chris's suggestion from 2015, that the CFT is not running in the same flow control as the normal code nodes on the Unix box.  As we can't select a server and can run code rather than it be queue, it looks like CFT is running in the Windows environment and is totally unrelated to what the SAS connections are doing.

Doesn't solve my problem, but I feel it helps my understanding.

[PhilGUK]

As this seems to be successful the 2nd and subsequent time of asking, I'm wondering whether putting in a fake CFT that I want to fail earlier in the process will allow the one I really want to succeed?

[ChrisHemedinger]

Hi @PhilGUK ,

Looks like you've been iterating on this rapidly, and getting close to a solution. All I can say is that from an EG flow standpoint, adding that user-defined link (as you did) should enforce the sequence and the Copy task won't run (theoretically) until the prior tasks/connected branch completes. If you find that's not happening, you can open a case with Tech Support. They will probably ask you for an Application Log (generated from EG), as that has a lot of detail that might help.

[PhilGUK]

Thank you for coming back so quickly. I haven't tried putting in a throw away CFT at the start, as now I've run it once, I can't replicate. Something for tomorrow and I will consider the Tech Support route (once I figure out how to save the Application Log).

[PhilGUK · Accepted solution]

SOLVED!!
I have found a work around that seems reliable.  I don't really get why it works.  I raised a ticket with SAS Support and as I tested various things, it is clear (and well understood by SAS) that the Copy Files Task runs under the Windows Client level "it does look like a timing issue where macro's are defined in the Unix platform, but the CFT task is at the Windows client level".
This is my modified flow:

I found that if I ran this stream from Execute_BVR, it would always fail.  If I ran it from Export to Excel Unix, it would always work.
After a bit of trial and error, I inserted another code node - Dummy to make Copy Files Task Work! which has no code in it (just a comment to say why it is there).  The flow seems to work every time (provided the extra node was either side of Export, it did not work if before Execute).

This is an submission status without the extra code node

The CFT takes 0.025s and fails.

And this one with the additional code

Where the CFT now takes 3.2s and even though the "Delete" starts 1s later, all is well.  The file is copied to Windows and has been removed from Unix.