Multi Processing

[meighanj]

I have code which splits a task into a number of parts and assigns a remote server to a part of this work.

%macro mtp_submit(_task_id);

    %macro Dummy; %mend Dummy;

    %syslput rem_proc_id=&_task_id. / remote=task&_task_id.;

    rsubmit task&_task_id. wait=no;
        /* code for parallel task */

        %logistic_models(
            comb_data      =MMODL.VAR_COMB8
            _proc_id       =&rem_proc_id.
        );

    endrsubmit;

%mend mtp_submit;


%macro mtp_gener_n(_task_cnt);
    %macro Dummy; %mend Dummy;

    %do idx=1 %to &_task_cnt.;
    
        %mtp_submit(&idx.);

    %end;

    waitfor _all_ 
    %do idx=1 %to &_task_cnt.;
        task&idx.
    %end;
    ;

    signoff _all_;

%mend mtp_gener_n;

The input file VAR_COMB8 has a column numbered from 1 to _task_cnt which specifies which remote session is to do each part of the work. 

The problem is that different sessions are taking significantly different lengths of time to complete their work. 

What I want to do is have each session complete a smaller piece of work, report back to the client that it has completed its work and be assigned more work to do until all work is done. 

I can't find anything in the documentation that would allow for this. 

Any suggestions would be great, 

Thanks

[PaigeMiller]

I see two issues, there may also be others

First, I don't see what the macro %logistic_models is supposed to do.

Second, when you send code to a remote machine, it doesn't always begin to execute immediately. If one of the remote machines is heavily loaded, your code won't execute immediately, or will execute slowly, and so there's no guarantee that the different remote machines should perform the task in the same amount of time.

[meighanj]

The macro %locstic_models runs a number of logistic models. However, this question could generalise to any equivalent piece where work is completed by remote machines.

Your second point is the reason I needed to ask this question. Some of my tasks are taking significanlty longer time to complete their work than others. Splitting the workload up into smaller pieces and allowing some remote machines to perform a larger proportion of the work will allow for natural load balancing. 

[PaigeMiller]

One other though regarding logistic regressions, the solution is an iterative maximum likelihood solution and so different regressions could take noticeably longer than others (even if the number of data points used is the same), because the iteration converges slowly.

As far as the loading on the servers, I would think you might want to talk to the IT support at your site to see if there is some way of determining the loading on the servers when you run your code.