how slow would the TTest procedure be ?

[ballardw]

---

@wang267 wrote:

Thank you so much for the respond. Yes, for repeated analysis we tried to minimize the possible output that are not used. I already turned off all the graphics, notes, outputs ,as such 

ods graphics off;
ods exclude all;
ods noresults;
options nonotes;

however, the hours of running Ttest occurs after all these were tuned down!! I also notice the Reg procedure is also slow, but at least it runs, The ttest seems not moving at all, and the running time doubled each time for the next loop. And at a time before I noticed, it gave me error like this: 

7969     %stage_analysis_srs;
ERROR: No directory space left in ITEM STORE WORK.SASTMP-000000002
ERROR: No directory space left in ITEM STORE WORK.SASTMP-000000002
ERROR: No directory space left in ITEM STORE WORK.SASTMP-000000002

---

That sort of error message makes me believe that you are running in a server mode and your job has hit a storage limit set by the server administrator. Or possibly you have created more output files than the OS can create file handles in a single directory.

[Rick_SAS]

 Please post your code.  Please post your code. Please post your code.

For tips and a PROC TTEST example for simulated data, see "Using simulation to estimate the power of a statistical test" and the links in that article about using BY groups and suppressing output. 

In my book and on my blog I have shown examples of using simulation and PROC TTEST to compute power curves. Below is an example from my blog. It runs in about 5 seconds on my PC. Try it and report what happens.

/* Program to use simulation to compute te power of the t test 
   for mean differences in (0 to 2 by 0.1) */
%macro ODSOff(); /* Call prior to BY-group processing */
   ods graphics off;
   ods exclude all;
   ods noresults; options nonotes;
%mend;
 
%macro ODSOn(); /* Call after BY-group processing */
   ods graphics on;
   ods exclude none;
   ods results; options notes;
%mend;
 
%let n1 = 10;           /* group sizes*/
%let n2 = 10;
%let NumSamples = 1000; /* number of simulated samples */  

data PowerSim(drop=i);
call streaminit(321);
do Delta = 0 to 2 by 0.1;
   do SampleID = 1 to &NumSamples;
      c = 1;                            /* Group 1 */
      do i = 1 to &n1;
         x = rand("Normal", 0, 1);      /* x ~ N(0,1) */
         output;
      end;
      c = 2;                            /* Group 2 */
      do i = 1 to &n2;
         x = rand("Normal", Delta, 1); /* x ~ N(Delta, 1) */
         output;
      end;
   end;
end;
run;

/* 2. Compute (pooled) t test for each sample */
%ODSOff 
proc ttest data=PowerSim; 
   by Delta SampleID; 
   class c; 
   var x; 
   ods output ttests=TTests(where=(method="Pooled")); 
run; 
%ODSOn
 
/* Construct indicator var for obs that reject H0 at 0.05 significance */ 
data Results; 
   set TTests; 
   RejectH0 = (Probt <= 0.05); 
run;

/* 3. Compute proportion: (# that reject H0)/NumSamples and CI */ 
%ODSoff
proc freq data=Results; 
   by Delta;
   tables RejectH0 / nocum binomial(level='1');
   output out=Est binomial;
run;
%ODSOn


/* display a power curve */
title "Power of the t Test";
title2 "Samples are N(0,1) and N(delta,1), n1=n2=10";
proc sgplot data=Est noautolegend;
   scatter x=Delta y=_BIN_ / yerrorlower=L_Bin yerrorupper=U_Bin;
   inset ("Number of Samples"="&NumSamples") / border;
   yaxis min=0 max=1 label="Power (1 - P[Type II Error])" grid;
   xaxis label="Difference in Population Means" grid;
run;

[wang267]

@Rick_SAS

Dr. Wicklin, 

Thank you so much for responding. I am running your code, but obviously it did not finish in 5 seconds. I took about 3 mins as I timed it. Again most of the time it shows "PROC TTEST running", even though in fact it might be other procedure running. 

So, there seems to be a problem with my computer, or SAS on my computer. I just restarted computer this Monday, and SAS earlier today. But I have access to 7 computers in all and have the program running on all of them. Now all computers get slow down, and is getting increasingly slow. Did my program make them slow? What might be the problem? How to do now? 

[Rick_SAS]

Based on the code you have shown, I suspect that you have filled up disk space, work directories, or exhausted some resource. Exit out of SAS, reboot your computer, and then restart SAS and run my program. My computer is about 7-8 years old, so it is not state-of-the-art.

[wang267]

@Rick_SAS

Thank you. Yes , I believe I filled them up, too, even though I am not quite sure how I filled. Is there a way to clear those spaces, directory and resources once in a while with code, to prevent those from being filled up again? 

[wang267]

I reboot the computer and run your program. Again the TTEST runs for 3 mins. I am using a desktop with 32 GB RAM, and over 200 GB disk space left 

[Reeza]

You had error messages previously that indicate you’re writing to a network or server. 

Use the the binary method to debug your code. Split it in half and test the first half and then second half. If you can’t split it, start running each one proc at a time to the macro and keep only the output you need. Look into ODS SELECT/EXCLUDE. 

---

@wang267 wrote:

I reboot the computer and run your program. Again the TTEST runs for 3 mins. I am using a desktop with 32 GB RAM, and over 200 GB disk space left 

---

[FreelanceReinh]

---

@Rick_SAS wrote:

In my book and on my blog I have shown examples of using simulation and PROC TTEST to compute power curves. Below is an example from my blog. It runs in about 5 seconds on my PC. Try it and report what happens.

---

Thanks a lot, @Rick_SAS, for chiming in. Now I'm really curious: How did your PC manage to run that program "in about 5 seconds"? My workstation (high-end hardware from 2015) takes consistently about 150 seconds for the PROC TTEST step alone (the remaining steps add almost nothing). This is comparably close to what @wang267 has reported (3 min).

I've checked/modified various system options (THREADS, CPUCOUNT, MEMSIZE, REALMEMSIZE, MAXMEMQUERY, MEMBLKSZ, MEMMAXSZ), to no avail.

Surprisingly (to me), the relationship between &NumSamples (and hence the number of BY groups) and run-time seems to be non-linear: With half the number of BY groups (&NumSamples=500) the step runs more than twice as fast: I've observed run-times as short as 32 seconds (rather than 150/2=75), but they tend to increase considerably in repeated runs (in the same SAS session), e.g.: 32, 49, 57, 87, 76, 111, 129, 139, 128 s. Each of these runs adds 24.2 MB (about 3.7 times the size of input dataset PowerSim) to the size of a "SAS Item Store" sastmp-000000002.sas7bitm in the WORK library. These three observations are indeed consistent with @wang267's problem description.

[Reeza]

@FreelanceReinh Mine also ran in about 5 seconds or less. 

I'm using SAS 9.4 TS1M5 Base on a Windows 7 machine with 256GB and 4 GB of RAM. 

[FreelanceReinh]

Hi @Reeza, thanks for testing. So, you confirm the speed factor of 30 (5 vs. 150 s). How strange!

I've just looked through the relevant "What's new" documents (my SAS 9.4 release is TS1M2 with SAS/STAT 13.2), but for PROC TTEST they mention only the new BOOTSTRAP statement.

One more finding: With the WORK library on a conventional hard disk, the TTEST step with NumSamples=500 is almost three times slower (>90 s vs. 32 s) than with my usual configuration, i.e. WORK on a RAM disk / fast SSD. Loading the input dataset into RAM (using the SASFILE statement) doesn't reduce the run-time. So, it seems that the I/O involving the .sas7bitm file is more relevant.

[wang267]

@FreelanceReinh: 

Yes, what @FreelanceReinh experienced about the increased time consumption for the same procedure in the same SAS session (i.e. without terminating SAS) is consistent with my problem. My first loop spend less than 30 mins, but started to be 2 hr, 3 hr, 5hr, 7 hr, unfinishable afterwards. And I am afraid this problem won't not be solved by simple restart SAS or even restart the computer.  After another try last night, which is of course unfinished in the whole night, now the disk became like this: 

I dont know what what happened and why this happen. 

[Reeza]

1. Find your DISER library and delete any data sets not needed there.

2. Open a new SAS session, right click the work library and find the path. Go that folder and then up a level. See how many SAS temp data sets are there and remove any not needed. If SAS crashed and you had datasets in the work library they are not deleted and take up space on your machine.

3. Add some PROC DATASETS to your macros to delete data sets when you don't need them to avoid issues with your loops/multiple calls and just keeping them around. SAS does do automatic clean up at the end of a session but not cleaning up after yourself can mess things up pretty good, especially when doing macro simulations. For example, if a proc fails for some reason, say PROC LOGISTIC, but you have the output from the previous step hanging around, the next step will just use your data set called A and then your results are all off. 

---

@wang267 wrote:

@FreelanceReinh: 

Yes, what @FreelanceReinh experienced about the increased time consumption for the same procedure in the same SAS session (i.e. without terminating SAS) is consistent with my problem. My first loop spend less than 30 mins, but started to be 2 hr, 3 hr, 5hr, 7 hr, unfinishable afterwards. And I am afraid this problem won't not be solved by simple restart SAS or even restart the computer.  After another try last night, which is of course unfinished in the whole night, now the disk became like this: 

I dont know what what happened and why this happen. 

---

[wang267]

Thanks @Reeza. Those are good suggestions. I just checked all these places, but not many items are found there. I expect to have many garbage in the Work library, but to my disappointment only 4 items, largest 96 KB are found there. 

[Reeza · Accepted solution]

Press Windows + F to bring up the Windows Search.

In the search, type in Size:Gigantic and see what pops up. Sadly their sizes are pretty small. 

https://www.dummies.com/computers/pcs/how-to-locate-large-files-on-a-hard-drive-using-windows-7/

[wang267]

I did as @Reeza suggested and freed the space. So it might be that my data sets (maybe middle step data sets from the analysis) are just too large. Let me try a half size.