topic Re: processing submitted statements after running regression macro in SAS Programming

processing submitted statements after running regression macro

doreamonjin — Tue, 10 Apr 2018 03:00:46 GMT

After I run the macro below, it will always take around 20 min to deal with 2 or more independent variables(vlist).

%macro fmreg(dvar=, vlist=, bvar=, inputdata=, outputname=);
data FMRegData;
set &inputdata; nmiss=nmiss(of &dvar &vlist);
keep &dvar &vlist &bvar nmiss mvlag;

data FMRegData;
set FMRegData;
where nmiss eq 0 ; run;

proc sort data=FMRegData; by &bvar; run;

ods select none;
proc glm data=FMRegData;
        ods output ParameterEstimates=Parm(drop=Dependent Probt Stderr rename=(Parameter=Variable)) fitstatistics=fitstatistics;
by &bvar;
        model &dvar=&vlist / solution;
  weight mvlag;
run; quit;

%WINSORIZE ( INSET=Parm , OUTSET= Parm , SORTVAR= Variable ,VARS= Estimate ,PERC1= 10,TRIM=0);

proc summary data=Parm nway;
class Variable;
output out=Parm2(drop=_type_ rename=(_freq_=N)) mean(Estimate TValue)=estimates MeanT Std(Estimate)=STD;
run;
data clus2dstats2(rename=(PARAM=estimates));
set Parm2;
tstat=(estimates*sqrt(N-1))/Std;
if abs(tstat) ge 1.645 then p='* '; if abs(tstat) ge 1.960 then p='** '; if abs(tstat) ge 2.576 then p='***';
est=put(estimates, 12.7); param=est; PARAM=compress(est||p); drop estimates;
run;

data coeff; set clus2dstats2(keep=variable estimates); run;

data tstat(drop=tstat rename=(tstat2=estimates)); set clus2dstats2(keep=variable tstat); tstat2=put(tstat, 12.3); tstat2=compress('('||tstat2||')')/*('('||left(tstat2)||')')*/; run;

proc summary data=fitstatistics nway;
output out=FFFit(drop=_type_ _freq_) mean(RSquare)=Result; run;

data FFIT2(keep=Variable Estimates); retain Variable; set FFFIT; Variable='RSQ'; Result=Result*100; nvalue2=put(Result,12.7); estimates=left(nvalue2); run;

data both(rename=(Estimates=Est_&outputname));
set coeff tstat FFIT2;
run;

proc sort data=both out=both_&outputname; by Variable; run;

%mend fmreg;

Re: processing submitted statements after running regression macro

PGStats — Tue, 10 Apr 2018 03:20:32 GMT

Try adding PLOTS=NONE to the proc glm statement if you don't need the graphs. Plotting consumes a lot of resources sometimes.

Re: processing submitted statements after running regression macro

doreamonjin — Tue, 10 Apr 2018 03:40:15 GMT

thank you so much, and you really do me a big favour

Re: processing submitted statements after running regression macro

doreamonjin — Tue, 10 Apr 2018 09:50:30 GMT

But now if I run several macros togehter, still needs to wait for a long time. Could you help me to figure it out again? Thank you

Re: processing submitted statements after running regression macro

ballardw — Tue, 10 Apr 2018 18:45:11 GMT

@doreamonjin wrote:
But now if I run several macros togehter, still needs to wait for a long time. Could you help me to figure it out again? Thank you

You are hiding details of what you are doing with the macro %winsorize

This reads data twice when not needed:

data FMRegData;
set &inputdata; nmiss=nmiss(of &dvar &vlist);
keep   &dvar &vlist &bvar nmiss mvlag;

 data FMRegData;
 set  FMRegData;
 where nmiss eq 0 ; run;

Could be

data FMRegData;
   set &inputdata; 
   nmiss=nmiss(of &dvar &vlist);
   keep   &dvar &vlist &bvar nmiss mvlag;
   if nmiss eq 0 ; 
run;

Warning: habitual use of the

data olddataset;

set olddataset;

will at some time cause you significant time when you have a code error and replace values in a data set that are potentially unrecoverable.

You might try setting fullstimer option before running the code and see what pieces are consuming time, which will probably work best with options Mprint as well.

How big (number of records) is your input data set? How many levels of the by variables?

Re: processing submitted statements after running regression macro

Reeza — Tue, 10 Apr 2018 14:53:20 GMT

1. Combine your first two data steps into one step

2. Format your code so it's legible and comment it as well - This makes it easier to see where steps can be combined.

3. Same with the last set of data steps, combine as many as possible, you may need to re-order the steps. Remember you can rename/add/drop on the SET statement or with PROC DATASETS so if you don't need to process the data do not use a data step.

4. Turn off ODS destinations so that no output is generated and that should save you some more time.

@doreamonjin wrote:

After I run the macro below, it will always take around 20 min to deal with 2 or more independent variables(vlist).

%macro fmreg(dvar=, vlist=, bvar=, inputdata=, outputname=);
data FMRegData;
set &inputdata; nmiss=nmiss(of &dvar &vlist);
keep   &dvar &vlist &bvar nmiss mvlag;

data FMRegData;
set FMRegData;
where nmiss eq 0 ; run;

proc sort data=FMRegData; by &bvar; run;

ods select none;
proc glm data=FMRegData;
        ods output ParameterEstimates=Parm(drop=Dependent Probt Stderr rename=(Parameter=Variable)) fitstatistics=fitstatistics;
by &bvar;
        model &dvar=&vlist / solution;
  weight mvlag;
run; quit;

%WINSORIZE ( INSET=Parm , OUTSET= Parm , SORTVAR= Variable ,VARS= Estimate ,PERC1= 10,TRIM=0);

proc summary data=Parm nway;
class Variable;
output out=Parm2(drop=_type_ rename=(_freq_=N)) mean(Estimate TValue)=estimates MeanT Std(Estimate)=STD;
run;
data clus2dstats2(rename=(PARAM=estimates));
set Parm2;
tstat=(estimates*sqrt(N-1))/Std;
if abs(tstat) ge 1.645 then p='* '; if abs(tstat) ge 1.960 then p='** '; if abs(tstat) ge 2.576 then p='***';
est=put(estimates, 12.7); param=est; PARAM=compress(est||p); drop estimates;
run;

data coeff; set clus2dstats2(keep=variable estimates); run;

data tstat(drop=tstat rename=(tstat2=estimates)); set clus2dstats2(keep=variable tstat); tstat2=put(tstat, 12.3); tstat2=compress('('||tstat2||')')/*('('||left(tstat2)||')')*/; run;

proc summary data=fitstatistics nway;
output out=FFFit(drop=_type_ _freq_) mean(RSquare)=Result; run;

data FFIT2(keep=Variable Estimates); retain Variable; set FFFIT; Variable='RSQ'; Result=Result*100; nvalue2=put(Result,12.7); estimates=left(nvalue2); run;

data both(rename=(Estimates=Est_&outputname));
set coeff tstat FFIT2;
run;

proc sort data=both out=both_&outputname; by Variable; run;

%mend fmreg;