SAS DI: fork trf in job does not work when job is scheduled

[mvanmun]

Hi there,

we use fork transformations in load jobs to enable parallel processing. These jobs work fine when we start the job in DI. All forks are executed.

However after scheduling the jobs in MC only the first fork is processed or nothing is processed at all. Moreover, the load jobs don't fail, but end "successfully"...

We have no clue what we are doing wrong. Is there some kind of setting we missed? Any advice is more than welcome!

System info

=================

DI version: 4.902

Meta data server: 9440

OS: Win 7.6.2

Thx.

Marcel

[Patrick]

@mvanmun 

How do you know that the forks haven't been executed?

Reason I'm asking: I've been diving lately in detail into the loop transformation generated code (the fork generated code is very similar). At least in the loop transformation the status of "Finished" for the loop is based on the return code of the inner job.

For the outer job to finish successfully: If run in parallel (so same as the Fork does) then the outer job would end with errors in case the signon or rsubmit doesn't succeed. The generated code from the Fork is to my knowledge identical in these parts.

[Patrick]

@mvanmun 

Just had another idea what could be happening here.

The forked process runs as a asynchronous child process. Once started control is given back immediately to the parent process (your DIS job). Make sure to also use transformation "Wait for completion" and to connect all forked processes to such a transformation. This will ensure that the parent process is waiting at this point for the child processes to finish.

So what could be happening: You're not using the "Wait for completion" transformation. Child processes get started and then the main process continues running to the end without ever waiting for the children to finish. Running out of DIS you've got an ongoing session established (the parent) so the children can still finish. In batch the parent session will end once it's done all of its execution. If it doesn't wait for the children to end then I guess any child process not yet finished will get killed when the parent process ends. 

I'm only making an educated guess here but this would explain what you observe and why the results are different when run out of DIS as compared to run in batch. 

[mvanmun]

Hi Patrick,

Thx for your input, highly appreciated! 

First of all, we know that the forks don't work since we don't see any entries in target table. 

Secondly, we had similar thoughts on the Wait for completion transformation and added this trf to the load job, but no success.

So … still struggling. Keep you posted. 

cheers Marcel

[Patrick]

"we know that the forks don't work since we don't see any entries in target table"

I'd analyze the SAS Log. May be search for the locations where Signon and Rsubmit get executed. If the Fork doesn't do something different from the Loop here then an unsuccessful Signon or Rsubmit should lead to your job to end with errors.

If the Signon/Rsubmit succeed the you should also get somewhere a SAS log with the handle name as part of the log name and you could inspect this log to better understand what's not working.