Hi ,
I have encountered memory issues while running proc means procedure on a dataset. I am getting below warning when I run my program and the generated P statistics are set to missing.
proc means data=mainlib.pla_app_ecdm(keep=V1 V2 V3) noprint;
var V1 V2 V3;
output out=DMWORK.meanout1(drop =_freq_ _type_)
N= N_1 N_2 N_3
NMISS= NMISS_1 NMISS_2 NMISS_3
MEAN= MEAN_1 MEAN_2 MEAN_3
STD= STD_1 STD_2 STD_3
MIN= MIN_1 MIN_2 MIN_3
P1= P1_1 P1_2 P1_3
P5= P5_1 P5_2 P5_3
P10= P10_1 P10_2 P10_3
P25= P25_1 P25_2 P25_3
P50= P50_1 P50_2 P50_3
P75= P75_1 P75_2 P75_3
P90= P90_1 P90_2 P90_3
P95= P95_1 P95_2 P95_3
P99= P99_1 P99_2 P99_3
MAX= MAX_1 MAX_2 MAX_3
;
run;
NOTE: The affected statistics will be missing from the corresponding classification levels.
WARNING: A shortage of memory has caused the quantile computations to terminate prematurely for QMETHOD=OS. Consider using
QMETHOD=P2.
NOTE: The affected statistics will be missing from the corresponding classification levels.
NOTE: There were 3911034 observations read from the data set MAINLIB.PLA_APP_ECDM.
NOTE: The data set DMWORK.MEANOUT1 has 1 observations and 90 variables.
NOTE: Compressing data set DMWORK.MEANOUT1 increased size by 100.00 percent.
Compressed is 2 pages; un-compressed would require 1 pages.
NOTE: PROCEDURE MEANS used (Total process time):
real time 28.90 seconds
cpu time 1:19.52
- What could be an issue? I have around 2 TB free space under Unix box.
- Does it require more RAM space? Not sure about the available RAM.
- Is there anyway, I can calculate the memory required for the processing and then decide if I should switch the method to P2?
- Are there any other procs that I can use to get rid of this warning and still get the right stats?
Appreciate your help on this.
Thanks,
Kulkarnj
Just compute Percentile and leave out others ? one variable one time .
OR could try proc ranks .
According to the documentation the OS (order-statistics) method requires the entire data set to fit into memory.
You can use a different method or give your job more memory.
Thanks all!
Proc means definitely fails when it runs out of memory. Other way to use P2 method which approximates the stats. However, I wanted to use P2 when it runs out of memory otherwise OS method. Another challenge was to guess when the proc means would fail (My process is completely automated).
I decided to use proc stdsize just for quantile calculations, as this proc is smart enough to switch the method to P2 when there is no enough memory to estimate quantiles. But I had to use proc means for other stats (Mean, STD Max etc) and combine the results.
Any comments?
Thanks,
JK
An alternative way is using IML .
SAS Innovate 2025 is scheduled for May 6-9 in Orlando, FL. Sign up to be first to learn about the agenda and registration!
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.
Ready to level-up your skills? Choose your own adventure.