data step: generating many subsets

[Lida]

Hi Astounding,

Thank you for great suggestions. I will follow your advice. It should work.

Thank you ALL for the input and have a great day. I will mark this question answered.

Lida

[art297]

Without being familiar with the input data, and not knowing what your included code is doing, one can only guess.

However, if all you need to do is calculate the individual variances, as well as the overall variance, why not just use proc summary?

e.g., it works quite quickly for a small file with 250,000 levels of the class variable:

data test;

  do bstr=1 to 250000;

    x=4;

    output;

    x=6;

    output;

    if bstr lt 15000 then do;

      x=8;

      output;

    end;

  end;

run;

proc summary data=test;

  class bstr;

  var x;

  output out=variance var=variance_x;

run;

[Reeza]

I think proc summary would only simplify the last portion of the code the OP wanted as the the variance required is the variance of the life expectancy estimates produced by the macro.

[Tom]

Not really any less efficient if you make the index first.

You will really only be reading the large file twice.

1) Creatinge the index on BSTR.

2) Creating the individual files will use the index so each disk block of the 1.5 gig file will be read only once or twice.

[Lida]

I thought about using an INDEX too. Will try. Thank you Tom for the advice!

[Haikuo]

I don't know the up limit of how many tables you can generate on one pass. But I concur with Tom. You should be able to solve your problem by using his suggestion. Of course, it may not as efficient as your original plot though.

just my 2 cents,

Haikuo