BookmarkSubscribeRSS Feed
kulkarnj
Calcite | Level 5

Hi ,

I have encountered memory issues while running proc means procedure on  a dataset. I am getting below warning when I run my program and the generated P statistics are set to missing.

proc means data=mainlib.pla_app_ecdm(keep=V1 V2 V3) noprint;

  var V1 V2 V3;

  output out=DMWORK.meanout1(drop =_freq_ _type_)

  N= N_1 N_2 N_3

  NMISS=   NMISS_1 NMISS_2 NMISS_3

  MEAN=   MEAN_1 MEAN_2 MEAN_3

  STD=   STD_1 STD_2 STD_3  

  MIN=   MIN_1 MIN_2 MIN_3

  P1=   P1_1 P1_2 P1_3 

  P5=   P5_1 P5_2 P5_3 

  P10=   P10_1 P10_2 P10_3 

  P25=   P25_1 P25_2 P25_3 

  P50=   P50_1 P50_2 P50_3  

  P75=   P75_1 P75_2 P75_3 

  P90=   P90_1 P90_2 P90_3 

  P95=   P95_1 P95_2 P95_3  

  P99=   P99_1 P99_2 P99_3  

MAX=   MAX_1 MAX_2 MAX_3 

;

run;

NOTE: The affected statistics will be missing from the corresponding classification levels.

WARNING: A shortage of memory has caused the quantile computations to terminate prematurely for QMETHOD=OS. Consider using

         QMETHOD=P2.

NOTE: The affected statistics will be missing from the corresponding classification levels.

NOTE: There were 3911034 observations read from the data set MAINLIB.PLA_APP_ECDM.

NOTE: The data set DMWORK.MEANOUT1 has 1 observations and 90 variables.

NOTE: Compressing data set DMWORK.MEANOUT1 increased size by 100.00 percent.

      Compressed is 2 pages; un-compressed would require 1 pages.

NOTE: PROCEDURE MEANS used (Total process time):

      real time           28.90 seconds

      cpu time            1:19.52

- What could be an issue? I have around 2 TB free space under Unix box.

- Does it require more RAM space? Not sure about the available RAM.

- Is there anyway, I can calculate the memory required for the processing and then decide if I should switch the method to P2?

- Are there any other procs that I can use to get rid of this warning and still get the right stats?

Appreciate your help on this.

Thanks,

Kulkarnj

4 REPLIES 4
Ksharp
Super User

Just compute Percentile and leave out others ? one variable one time .

OR could try proc ranks .

data_null__
Jade | Level 19

According to the documentation the OS (order-statistics) method requires the entire data set to fit into memory.

You can use a different method or give your job more memory. 

kulkarnj
Calcite | Level 5


Thanks all!

Proc means definitely fails when it runs out of memory. Other way to use P2 method which approximates the stats. However, I wanted to use P2 when it runs out of memory otherwise OS method. Another challenge was to guess when the proc means would fail (My process is completely automated).

I decided to use proc stdsize just for quantile calculations, as this proc is smart enough to switch the method to P2 when there is no enough memory to estimate quantiles. But I had to use proc means for other stats (Mean, STD Max etc) and combine the results.

Any comments?

Thanks,

JK

Ksharp
Super User

An alternative way is using IML .

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 4 replies
  • 3407 views
  • 0 likes
  • 3 in conversation