About tingo

tingo · ‎02-06-2017

Thanks stat_sas and Rick_SAS for your suggestions. To try them, I downsized the data to a 5GB dataset with 250k obs and still 2500 vars. Now, executing the previous proc reg code without the output statement takes 18 seconds, while it takes aprox 2 minutes and 5 seconds with the output statement (requesting both rstudent and h), regardless of whether the out= dataset has or not the same name as the input data= dataset. So even at this data size there is a significant increase in the computational time required by the output statement. To further explore whether this increase is caused by the computation of studentized residuals and leverages or by the I/O, I tried different output statements: 1) output out=training1 h=h; Took aprox 2 minutes 2) output out=training1 rstudent=rstud; Also took 2 minutes 3) output out=training1 (keep=rstud h) rstudent=rstud h=h; Took 1 minute and 50 seconds 4) output out=training1; Took 45 seconds Using the INFLUENCE option on the code without the OUTPUT statement made no difference since the NOPRINT option was on. But I think the comparison would not be fair removing NOPRINT as it then would take a long time writing influence statistics to the results window (250k lines of 2500 dfbetas plus the other statistics...). Anyway, the comparison between the output statements 1-4 above seems to point to the computation of the leverages and residuals taking the biggest part of the time - it took just 25-30 seconds more (wrt the code without output statement) to produce the whole out= dataset (with the 2500 vars), but aprox 1 minute and a half more to compute the h and rstudent options and write them down to the out= dataset (even just these 2 columns). It looks like it is the computation of the H matrix diagonal (which is not actually needed in the stepwise selection process but is necessary for studentized residuals and leverages) what makes the difference in elapsed time. I will appreciate any comments on these findings as well as further ideas to obtain studentized residuals in a shorter time (perhaps proc IML...?) Thanks again!

tingo · ‎02-05-2017

Hi, while running different regression analyses on a 17GB dataset (aprox. 900k instances and 2500 variables) using PROC REG, I realized there is a huge difference in terms of time-to-result between the following two programs: Without Output statement: This takes aprox 1 minute to conclude. proc reg data=training outest=pars noprint; by v1; /*this by-variable has only 2 levels*/ model vm=&vars / selection=stepwise adjrsq aic include=1 sle=0.001 sls=0.005 ; /*&vars contains the 2500 variables*/ weight invvar ; run; quit; With Output statement: This, however, takes aprox 1 hour to conclude. proc reg data=training noprint; by v1; /*this by-variable has only 2 levels*/ model vm=&vars / selection=stepwise adjrsq aic include=1 sle=0.001 sls=0.005 ; /*&vars contains the 2500 variables*/ weight invvar; output out=training rstudent=rstud h=h ; run; quit; Notice that the only differences are that in the first code a outest= dataset (pars) is required but there is not an Output statement, while in the second one an output statement is included but the outest= dataset is not required. Why just including the Output statement mades such a big difference? Writing data to the out= dataset alone should not account for such a difference, since the data step that builds the 17GB training dataset takes just 5 minutes to conclude. May it be the computation of studentized residuals and leverages what takes so long? By the way, how would you shorten the time required to produce such an out= dataset? Thanks!

Online Status	Offline
Date Last Visited	‎02-07-2017 02:52 PM

Re: PROC REG Output statement takes too long

PROC REG Output statement takes too long

Re: PROC REG Output statement takes too long

Re: PROC REG Output statement takes too long

PROC REG Output statement takes too long