topic Re: Making work queues of equal sizes in SAS Programming

Making work queues of equal sizes

DASDguy — Thu, 05 Oct 2017 16:32:19 GMT

Explanation:

SAS 9.4 m4

I've got about 100 pieces of work to accomplish. I know what the average run time is for each piece of work. I have only 4 connections to my database and I want to make the work run optimally. I tried proc hpbin but it does not do what I really want, but does give you an idea of my data.

proc hpbin data=work.hbin_data output=work.binout numbin=4 bucket;
id job_nm jobwall_p95;
input jobwall_p95;
run;

I then write a proc summary to see how it did and I have 4 queues, but they are not close to being equal as that not what HPBIN does.

BIN JOBWALL_P95_Sum JOBWALL_P95_N
1 378.66666667 77
2 129.81666667 3
3 173.71666667 2
4 132.06666667 1

So, HPBIN makes 4 bins with the first one that runs small jobs and takes 378 seconds

Please help?

Re: Making work queues of equal sizes

ballardw — Thu, 05 Oct 2017 16:48:39 GMT

If the issue is to assign your jobs into 4 bins then sort the data by the runtime and then apply a bin number using MOD function based on order in the data set.

The below example creates 100 records in sorted order. Think of the value as runtime order. The mod function would return 0 to 3 with mod 4, so I add 1 in case 0 isn't an acceptable bin number. You could sum the runtimes with the bin as a class variable to see the expected result.

data have;
   do i=1 to 100;
      output;
   end;
run;

data want;
   set have;
   bin = mod(_n_,4)+1;
run;

If you have SAS/OR licensed I suspect that you could get a better balance but I suspect this approach may be "good enough" unless you have one or two jobs that run orders of magnitude longer than most of the others.

Re: Making work queues of equal sizes

DASDguy — Thu, 05 Oct 2017 20:10:39 GMT

My problem is that some jobs take .06 seconds and other take over 100. So, there is quite some disparity there. I don't have SAS/OR. 😞

Re: Making work queues of equal sizes

ballardw — Thu, 05 Oct 2017 22:12:21 GMT

@DASDguy wrote:

My problem is that some jobs take .06 seconds and other take over 100. So, there is quite some disparity there. I don't have SAS/OR. 😞

On second though the mod approach, will tend to have the 4th bin run longer.

Consider:

data junk;
   incr= 1;
   do i = 1 to 100;
      bin+incr;
      if bin=5 then do;
         incr= -1;
         bin=4;
      end;
      if bin=0 then do;
         incr= 1;
         bin=1;
      end;
      output;
   end;
run;

proc means data=junk nway sum;
   class bin;
   var i;
run;

Re: Making work queues of equal sizes

DASDguy — Mon, 09 Oct 2017 18:49:27 GMT

The method of "snake drafting" is good enough for my purposes.