topic Re: How do I debug an infinite running proc survey select? in SAS Programming

How do I debug an infinite running proc survey select?

jamesf — Sun, 21 Jul 2019 09:01:15 GMT

I am trying to do a PPS proc survey select on a large number of records (~10 million) and am running into never-ending task runs.

The code works when I limit the input to particular sub-samples, but goes into a never-ending task on other sub-samples, which requires a full SAS reset. I am unsure how to get any debugging information from cases with never-ending tasks.

If I limit my input data to state 3 (200,000 units, 9,000 selections, 58 second run time). If I limit my input data to state 1(2,500,000 units, 140,000 selections), the code runs forever (3+ hours) until I force terminate it.

Any assistance as to how to start narrowing down the cause would be greatly appreciated.

My data is of the form

Data set: nh_p_stra

STATE SampleSize

1 2000

2 3000

3 4000

Data set: data_in

Unit_id STATE weight

0000001 1 1.2

0000002 1 2.1

0000003 1 0.9

0000004 2 0.8

0000005 2 1.3

My code is of the form

proc surveyselect method=pps data=data_in sampsize=nh_p_stra out=selection_out seed=123;

strata state;

id unit_id;

size weight;

run;

Re: How do I debug an infinite running proc survey select?

ballardw — Mon, 22 Jul 2019 15:49:04 GMT

I suspect that you are just terminating a tad too soon. Depending on how you did that the output set should have been populated, how far did you get?

Or run a loop selecting each state at a time, generating a separate output set then combine the results later.

Something like:

data _null_;
   length str $ 100.;
   do state= 1 to 50;
      call execute('proc surveyselect method=pps data=data_in sampsize=nh_p_stra');
      str=catt('(where=(state=',state,')) out=selection',state,'seed=123;');
      call execute (str);
      call execute('strata state;
         id unit_id;
         size weight;
         run;');
   end;
run;

I suspect the issue is just the size of your data set, as in numbers of records and possibly the values of your weight variable. There are going to be a lot of calculations, for instance the sum of weight per strata as a start, then calculate the proportion of that total each record represents. If you haven't read the documentation details on calculation PPS samples you may not realize the scope of computations involved in a data set this big

Re: How do I debug an infinite running proc survey select?

Watts — Mon, 22 Jul 2019 22:37:52 GMT

One factor to check is system options and resources. The METHOD=PPS alogorithm involves sorting. And sorting can become slow when system resources get low.

Also, consider whether a different selection method might be appropriate for the task. Some other types of PPS selection are generally faster -- e.g, systematic, sequential, with-replacement. In addition to equal-probability methods. (Doc is here.)

Another possibility is to make the strata smaller (by using additional stratification, or by simply subdividing them).

Re: How do I debug an infinite running proc survey select?

jamesf — Wed, 31 Jul 2019 07:18:29 GMT

Thanks for the help. Yeah, it turned out I terminated it too soon. It seems the run time goes up non-linearly with the size of the strata. I was able to get the code dialed in on the small states, before doing an overnight run on the larger states.