Multiply matrices by vectors, with replications

[avanwie]

I am using SAS 9.4

I have 15 teachers, each with three effectiveness estimates, one for math, one for reading and one for motivation. For each set of 15 teachers, there is a covariance matrix. I ran 1000 replications. I have two data sets, one with the 15 teachers for 1000 replications (15,000 rows, 3 columns) and one with the 3 by 3 covariance matrix for the 1000 replications (3,000 rows, 3 columns). 

I am trying to 'weight' each teacher's effectiveness estimate by the covariance matrix. So, each of the first 15 rows in the teacher data set is multiplied by the first 3x3 matrix. Then the next 16-30 rows in the teacher data set are multiplied by the second 3x3 matrix. and so on...

I don't think my code is iterating to the next set of 15 teachers. I think it is multiplying the same set of 15 teachers each time.

Here is my current code:

ODS select all;
Proc IML;
use TMP1.Covmatrices1 where (_NAME_ = "MMmathresid"|_NAME_= 'MMreadresid'|_NAME_= 'MMmotresid');
read all var {MMmathresid MMreadresid MMmotresid} into M[colname=varNames];
create MMMatrices1 from M [colname=varNames]; append from M; setout MMMatrices1;

use TMP1.residuals1;
read all var {MMmathresid MMreadresid MMmotresid} into N[colname=varNames];
create TchRes1 from N [colname=varNames]; append from N; setout TchRes1;

use residuals1;
read all var{Iter} into iter[colName=varNames];*print iter;
use residuals1;
read all var{IDT} into IDT[colName=varNames];*print IDT;
close;

/*must have 1x3 matrix times 3x3 matrix*/
T={1,1,1}; /*this multiplies by the product of MN and sums the elements to give one number*/
p=3; /*number of rows in M*/
k=nrow(N); print k;
o=1; /*number of rows in N*/
Na=N[o,]; print Na;
iters=countunique(iter); *print iters;
create iterss var {iter};append; setout iterss;
create idt var{IDT};append;setout idt;
tch=countunique(IDT); print tch;
CorrBasedNew2=j(k,1);/*allocates the variable for the product of MNT*/
create composites from CorrBasedNew2[c="Covcomposite"];

do j=1 to iters;/*loop over the number of replications*/
MstartRow=p*(j-1)+1 ; MendRow=MstartRow+p-1 ; print MstartRow;Print MendRow;

do i=1 to tch;/*loop over number of teachers so the same cov matrix is multiplied to all within this group*/
NstartRow=o*(i-1)+1; NendRow=NstartRow+o-1; print NstartRow;Print NendRow;

CorrBasedNew2=(N[NstartRow:NendRow,])*(M[MstartRow:MendRow,])*(T);

append from CorrBasedNew2;
end;
end;
close composites;
quit;

Note, there are some missing values in both sets- so I need to address that in the code as well. I tried adding this to the code:

if (N[NstartRow:NendRow,]) ^= . or (M[MstartRow:MendRow,]) ^= . then
CorrBasedNew2=(N[NstartRow:NendRow,])*(M[MstartRow:MendRow,])*(T); else CorrBasedNew2=0;

thanks, Anna

[Rick_SAS]

Your situation is too complicated for me to debug, so here is some general advice for developing and debugging your program:

1. Resolve the missing value issue. You can't do matrix multiplication with missing values.

2. Use 4 teachers, not 15

3. Use 5 replicates, not 1000.

4. Add a SampleID variable to the data sets. That will make it easier to keep track of which observations correspond to which sample. For example, in the covariance data, the first three observations have SampleID=1, the next three have SampleID=2, and so forth.

5. Include a DATA step so that others can run your program.

For large data sets, the most efficient way to perform "BY-group processing" in SAS/IML is to use the UNIQUEBY function. 

Because you know the number of obs in each data set, you don't necessarily need to use UNIQUEBY, but I mention it because the article has an example and discussion that might be relevant to your problem.