Appending dataset in Macro loop

[avanwie]

I am using SAS 9.4.

I am generating 3 outcomes (score+teacher residual+ student residual) for students within teachers. I have 22 students per teacher. There are 500 teachers. 

The student residual dataset is not appending for each student residual set per teacher. I expect to have 11,000 rows in the sturesids file but I only have 22. I can see that it is generating 22 students per teacher by using the print stu_res command. 

%macro generate;

%let nt = 500; /*number of teachers*/
%let nb = 22; /*number of students*/
%let n1=3; /*number of outcomes*/

%let nanb = &nt*&nb;

/*teacher cov - this a cov matrix from the corr matrix*/
%let tcov = {.25 .175 .0225,
.175 .25 .0225,
.0225 .0225 .0625};

/*student cov*/
%let scov = {1 .2 .0225,
.2 1 .0225,
.0225 .0225 1};

/*mean vector*/
%let mv= {0,0,0};

proc iml;
gmv={50 50 30}; /*grand mean vector*/

ynew=j(&nanb,&n1,0); /*this is for the new outcomes, should be as long as the stu*tch*/
y=j(&nb,&n1,0);
tch_res=j(&nt,&n1,0);
stu_res=j(&nb,&n1,0);

call randseed(0);

do A = 1 to &nt;
tch_res[A,]= (RandNormal(1,&mv,&tcov)); /* tchres matrix*/

do B = 1 to &nb;
stu_res[B,]= (RandNormal(1,&mv,&scov)); /*stures matrix*/ print stu_res;

do C = 1 to &n1;
if C = 1 then do;
y[B,1] = gmv[1,1] + tch_res[A,1] + stu_res[B,1];
end;
if C = 2 then do;
y[B,2] = gmv[1,2] + tch_res[A,2] + stu_res[B,2];
end;
if C = 3 then do;
y[B,3] = gmv[1,3] + tch_res[A,3] + stu_res[B,3];
end;
iter=&i;/*iter counter for outcome scores*/
itert=&i;/*iter counter for teacher residuals*/
iters=&i; /*iter counter for student residuals*/
end;

end;
yy = round(y,.01);
tn = &nb*(A-1)+1;
tend = &nb*A;
ynew[(tn:tend)`,1:&n1] = yy;
y=J(&nb,&n1,0);
iter=J(&nanb,1,&i);
itert=J(&nt,1,&i);
iters=J(&nb,1,&i);/*container for student iteration counter*/
end;

id1 = (1:&nb)`; *print id1;
idk = repeat (id1,&nt); *print idk;

id2 = (1:&nt)`;
idt = repeat (id2,&nb); *print id21;
call sort(idt,1); *print idt;

/*save tch resids and stu resids in files*/
tchres= tch_res||itert||id2;
stures= stu_res||iters||id1;

t = {mathtchresGen readtchresGen mottchresGen itert idt};
s= {e1 e2 e3 iters idk};
create tchresids from tchres [colname=t]; append from tchres;close tchresids;
create sturesids from stures [colname=s];append from stures; close sturesids; print stu_res;

ydone = idt||idk||ynew||iter;

/*create data set to use in proc mixed*/
c = {idt idk y1 y2 y3 iter};
create testdat from ydone [colname=c]; append from ydone; close testdat;
quit;

/*proc print data=testdat (obs=100); run; */

data tchresgens;
set tchresids;
run;

/*break up data so each generated outcome is a column*/
data newdat;
set testdat;
y=y1; wave=1; cwave=wave; output;
y=y2; wave=2; cwave=wave; output;
y=y3; wave=3; cwave=wave; output;

keep idt idk y wave cwave iter;
run;

%mend;
%macro runall;
%do i=1 %to 1; /*number of replications*/
%generate;
proc datasets;
append base=fulldata data=newdat force;/*generated outcomes by wave, all otucomes in one column*/
append base=fulltchresgen data=tchresgens force;
append base=fullsturesgen data=sturesids force;
append base=fulltestdat data=testdat force; /*generated outcomes by outcome_3 columns*/
%end;
%mend;
proc IML;
%runall;
quit;

[Rick_SAS]

You are creating and writing to the StuResids data set only once, and the APPEND statement is outside of any loop. 

I suspect you want to use the CREATE statement at the top of the PROC IML program and then put the APPEND statement inside one of the loops. There is an example in the article "Simulate many samples from a logistic regression model."

If I might suggest a few other improvements:

1. There is no need to use a macro loop. SAS/IML enables you to put the replication loop inside the program. You can append all the samples to the data sets without exiting IML and using PROC APPEND.

2. You are currently generating one MV Normal observation for each iteration of the "DO A =" loop. A more efficientimplementation would be to set

tch_res = RandNormal(&nt, &mv, &tcov);

and then loop over the rows of tch_res.

[avanwie]

Hi. Thank you for the feedback. 

I can't figure out where to put each append within the nested loops. I believe the code is creating the data I need. I'm just not getting the datasets. 

%let nt = 15; /*number of teachers*/
%let nb = 10; /*number of students*/
%let n1=3; /*number of outcomes*/

%let nanb = &nt*&nb;

/*teacher cov - this a cov matrix from the corr matrix*/
%let tcov = {.25 .175 .0225,
.175 .25 .0225,
.0225 .0225 .0625};

/*student cov*/
%let scov = {1 .2 .0225,
.2 1 .0225,
.0225 .0225 1};
/*mean vector*/
%let mv= {0,0,0};

proc iml;
gmv={50 50 30}; /*grand mean vector*/

y=j(&nb,&n1)||stu_res||iter||itert||iters;
tch_res=j(&nt,&n1);
stu_res=j(&nb,&n1);

itert=j(&nt,1); iters=j(&nb,1);

create OUT from y;
create tchres from tch_res;
create stures from stu_res;

call randseed(0);

tch_res= (RandNormal(&nt,&mv,&tcov)); /* tchres matrix*/ print tch_res;

do A = 1 to &nt; itert[A,]=A;

do B = 1 to &nb;
stu_res[B,]= (RandNormal(1,&mv,&scov)); /*stures matrix*/ print stu_res;
iters[B,]=B;

do C = 1 to &n1;

if C = 1 then do;
y[B,1] = gmv[1,1] + tch_res[A,1] + stu_res[B,1];
end;
if C = 2 then do;
y[B,2] = gmv[1,2] + tch_res[A,2] + stu_res[B,2];
end;
if C = 3 then do;
y[B,3] = gmv[1,3] + tch_res[A,3] + stu_res[B,3];
end;

print y;

end;
end;
end;

quit;

[Rick_SAS · Accepted solution]

I suspect that you are the only one who can determine where to put the append statements. We don't know what data sets you are trying to create. 

I see that you are trying to open (and write?) three different data sets. From a syntactic point of view, you might want to read about the SETOUT statement, which enables you make one of the open data sets ready for output. For example, here is one way to write your data. I have no idea if it is what you want:

  create OUT from y;
   create tchres from tch_res;
   create stures from stu_res;
   call randseed(0);
   tch_res=RandNormal(&nt, &mv, &tcov);
  setout tchres; append from tch_res;

   do A=1 to &nt;
      itert[A, ]=A;
      do B=1 to &nb;
         stu_res[B, ]=(RandNormal(1, &mv, &scov));
         /*stures matrix*/
         iters[B, ]=B;
         y[B, ]=gmv + tch_res[A, ] + stu_res[B, ];
         setout OUT; append from y;
      end;
      setout stures; append from stu_res;
   end;
   close OUT tchres stures;
   

[avanwie]

This is great. So much simpler than my code.

Thank you.