DATA Step, Macro, Functions and more

processing submitted statements after running regression macro

Reply
Occasional Contributor
Posts: 6

processing submitted statements after running regression macro

After I run the macro below, it will always take around 20 min to deal with 2 or more independent variables(vlist).

 

%macro fmreg(dvar=, vlist=, bvar=, inputdata=, outputname=);
data FMRegData;
set &inputdata; nmiss=nmiss(of &dvar &vlist);
keep   &dvar &vlist &bvar nmiss mvlag;

 data FMRegData;
 set  FMRegData;
 where nmiss eq 0 ; run;

proc sort data=FMRegData; by &bvar; run;

ods select none;
proc glm data=FMRegData;
        ods output ParameterEstimates=Parm(drop=Dependent Probt Stderr rename=(Parameter=Variable)) fitstatistics=fitstatistics;
 by &bvar;
        model &dvar=&vlist / solution;
  weight mvlag;
run; quit;

%WINSORIZE ( INSET=Parm , OUTSET= Parm , SORTVAR= Variable ,VARS= Estimate ,PERC1= 10,TRIM=0);

proc summary data=Parm nway;
 class Variable;
 output out=Parm2(drop=_type_ rename=(_freq_=N)) mean(Estimate TValue)=estimates MeanT Std(Estimate)=STD;
run;
data clus2dstats2(rename=(PARAM=estimates));
 set Parm2;
 tstat=(estimates*sqrt(N-1))/Std;
 if abs(tstat) ge 1.645 then p='*  ';  if abs(tstat) ge 1.960 then p='** ';  if abs(tstat) ge 2.576 then p='***';
 est=put(estimates, 12.7); param=est; PARAM=compress(est||p); drop estimates;
run;

data coeff; set clus2dstats2(keep=variable estimates); run;

data tstat(drop=tstat rename=(tstat2=estimates)); set clus2dstats2(keep=variable tstat); tstat2=put(tstat, 12.3); tstat2=compress('('||tstat2||')')/*('('||left(tstat2)||')')*/; run;

proc summary data=fitstatistics nway;
 output out=FFFit(drop=_type_ _freq_) mean(RSquare)=Result; run;

data FFIT2(keep=Variable Estimates); retain Variable; set FFFIT; Variable='RSQ'; Result=Result*100; nvalue2=put(Result,12.7);  estimates=left(nvalue2); run;

data both(rename=(Estimates=Est_&outputname));
 set coeff tstat FFIT2;
run;
 
 proc sort data=both out=both_&outputname; by Variable; run; 

%mend fmreg;

Esteemed Advisor
Posts: 5,626

Re: processing submitted statements after running regression macro

Posted in reply to doreamonjin

Try adding PLOTS=NONE to the proc glm statement if you don't need the graphs. Plotting consumes a lot of resources sometimes.

PG
Occasional Contributor
Posts: 6

Re: processing submitted statements after running regression macro

thank you so much, and you really do me a big favour
Occasional Contributor
Posts: 6

Re: processing submitted statements after running regression macro

But now if I run several macros togehter, still needs to wait for a long time. Could you help me to figure it out again? Thank you
Super User
Posts: 13,946

Re: processing submitted statements after running regression macro

[ Edited ]
Posted in reply to doreamonjin

@doreamonjin wrote:
But now if I run several macros togehter, still needs to wait for a long time. Could you help me to figure it out again? Thank you

You are hiding details of what you are doing with the macro %winsorize

This reads data twice when not needed:

data FMRegData;
set &inputdata; nmiss=nmiss(of &dvar &vlist);
keep   &dvar &vlist &bvar nmiss mvlag;

 data FMRegData;
 set  FMRegData;
 where nmiss eq 0 ; run;

Could be

 

data FMRegData;
   set &inputdata; 
   nmiss=nmiss(of &dvar &vlist);
   keep   &dvar &vlist &bvar nmiss mvlag;
   if nmiss eq 0 ; 
run;

 

 

Warning: habitual use of the

data olddataset;

    set olddataset;

will at some time cause you significant time when you have a code error and replace values in a data set that are potentially unrecoverable.

 

You might try setting fullstimer option before running the code and see what pieces are consuming time, which will probably work best with options Mprint as well.

 

How big (number of records) is your input data set? How many levels of the by variables?

Super User
Posts: 24,026

Re: processing submitted statements after running regression macro

Posted in reply to doreamonjin

1. Combine your first two data steps into one step

2. Format your code so it's legible and comment it as well - This makes it easier to see where steps can be combined.

3. Same with the last set of data steps, combine as many as possible, you may need to re-order the steps. Remember you can rename/add/drop on the SET statement or with PROC DATASETS so if you don't need to process the data do not use a data step.

4. Turn off ODS destinations so that no output is generated and that should save you some more time.

 


@doreamonjin wrote:

After I run the macro below, it will always take around 20 min to deal with 2 or more independent variables(vlist).

 

%macro fmreg(dvar=, vlist=, bvar=, inputdata=, outputname=);
data FMRegData;
set &inputdata; nmiss=nmiss(of &dvar &vlist);
keep   &dvar &vlist &bvar nmiss mvlag;

 data FMRegData;
 set  FMRegData;
 where nmiss eq 0 ; run;

proc sort data=FMRegData; by &bvar; run;

ods select none;
proc glm data=FMRegData;
        ods output ParameterEstimates=Parm(drop=Dependent Probt Stderr rename=(Parameter=Variable)) fitstatistics=fitstatistics;
 by &bvar;
        model &dvar=&vlist / solution;
  weight mvlag;
run; quit;

%WINSORIZE ( INSET=Parm , OUTSET= Parm , SORTVAR= Variable ,VARS= Estimate ,PERC1= 10,TRIM=0);

proc summary data=Parm nway;
 class Variable;
 output out=Parm2(drop=_type_ rename=(_freq_=N)) mean(Estimate TValue)=estimates MeanT Std(Estimate)=STD;
run;
data clus2dstats2(rename=(PARAM=estimates));
 set Parm2;
 tstat=(estimates*sqrt(N-1))/Std;
 if abs(tstat) ge 1.645 then p='*  ';  if abs(tstat) ge 1.960 then p='** ';  if abs(tstat) ge 2.576 then p='***';
 est=put(estimates, 12.7); param=est; PARAM=compress(est||p); drop estimates;
run;

data coeff; set clus2dstats2(keep=variable estimates); run;

data tstat(drop=tstat rename=(tstat2=estimates)); set clus2dstats2(keep=variable tstat); tstat2=put(tstat, 12.3); tstat2=compress('('||tstat2||')')/*('('||left(tstat2)||')')*/; run;

proc summary data=fitstatistics nway;
 output out=FFFit(drop=_type_ _freq_) mean(RSquare)=Result; run;

data FFIT2(keep=Variable Estimates); retain Variable; set FFFIT; Variable='RSQ'; Result=Result*100; nvalue2=put(Result,12.7);  estimates=left(nvalue2); run;

data both(rename=(Estimates=Est_&outputname));
 set coeff tstat FFIT2;
run;
 
 proc sort data=both out=both_&outputname; by Variable; run; 

%mend fmreg;


 

Ask a Question
Discussion stats
  • 5 replies
  • 206 views
  • 3 likes
  • 4 in conversation