Parallel processing in a Grid - sort of works

[LawrenceHW]

Hi all,

We recently moved to a SAS Grid and I am now getting started looking at parallel processing within programs (we already utilise the parallel submission of many independent programs via Grid Manager and that works well).

One of the programs I was looking to adapt is a program that does a series of PROCGENMOD using BY statements (which means they're independent) which was taking 10+ hours to run. After reading through a load of papers I found some solutions to the problem. I was able to cut the total run time down by 60% which is quite impressive.

The basic code I used is:

%macro ParallelProcess (TrtType=, Iteration=);
*-----------------------------------------------------------------------------*;
* use a macro do loop to iterate round each PARAMCD and within that loop      *;
* round each visit ID invoking RSUBMIT on each TASK node related to each visit.*;
*-----------------------------------------------------------------------------*;
  %do i=1 %to 10; %*** This loop allows 10 tasksto be in use ***;
    %let id=%eval(&i.+((&Iteration.-1)*10)); *** This will identify the source data to use in the model [values of 1 to 25]
    %syslput _global_/remote=task&i.;
    %syslput _local_/remote=task&i.;
    rsubmit task&i. wait=no connectpersist=yes inheritlib=(_tmpwork);
      %include "&Path2Area.\GenModInner.sas";
      %GenModInner;
    endrsubmit;
  %end;
  waitfor _all_;
%mend ParallelProcess;
%ParallelProcess (TrtType=placebo, Iteration=1);
%ParallelProcess (TrtType=placebo, Iteration=2);

%ParallelProcess (TrtType=placebo, Iteration=3);

signoff _all_;

This works pretty well but the problem I have with it is that even though 10 RSUBMIT tasks are going, it won't move onto the next set of models until all 10 tasks are completed. I realise this is to do with the macro invocation and the %DO loop I have defined but it's the only way I could get it working.

Does anyone know of any way of getting 10 RSUBMIT tasks open and each of the 25 independent process being submitted to the next available task until all 25 have finished and then signoff? I am limited to the number of RSUBMIT tasks and, potentially, we could have a lot more than 25 independent processes.

Thanks in advance for any idea.

Regards,

Lawrence

[tomrvincent]

I've had great success using our grid, but I use EG, not RSUBMITs I've had as many as 8 tasks running at the same time and the linking allows them to connect to subsequent tasks.

[LawrenceHW]

Hi Tom,

We require this to be executed in batch mode which is why I used RSUBMIT. I don't tend to use EG tasks as a rule (old school programmer - if it's not an ASCII file, it's not a SAS program ).

One of the other reasons I used RSUBMIT is that I don't know how many processes I will have to run as the project is ongoing so we needed something that could be data-centric.

Thanks for the swift reply though. I'll have to look into EG tasks as they're bound to prove useful for other programs.

Regards,

Lawrence