topic Re: Scaling for score residual in Statistical Procedures

Scaling for score residual

Tcook — Wed, 12 Dec 2012 13:12:18 GMT

Hi all,

I am trying to generate a residual plot for a weibull survival model based on the code used in "SAS Textbook Examples
Applied Survival Analysis by Hosmer, Lemeshow and May Chapter 8: Parametric Regression Methods" but I kept getting an error message about the matrices and I have tried all I could be to no avail. The fig84_a file in my own case has 9318 entries and 12 varaiables.

I would greatly appreciate if someone can take look at it and point out what exactly I need to do to fix it.

Below is my code and and error message received.

/*********code******************************/

proc iml;

use fig84_a ;

read all variables {lint ltype1 lsupply1 lperiod lru ltype2 luic lttename1 ltraname lspeed lspeed2 lscale} into L;

read all variables {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1 traname speed speed2 mgale} into X;

use covf84;

read all var _num_ into V;

dbeta = L*V;

ld = J(100, 1, 0);

do i = 1 to 100;

ld = L[i,]*t(dbeta[i,]);

end;

W = X || dbeta ||ld;

create fig84_b var {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1 traname speed speed2 mgale dbint dbtype1 dbsupply1 dbperiod dbru dbtype2 dbuic dbttename1 dbtraname dbspeed dbspeed2 dbscale ld};

append from W;

quit;

/***********ERROR MESSAGE************************

503 proc iml;

NOTE: IML Ready

504 use fig84_a ;

505 read all variables {lint ltype1 lsupply1 lperiod lru ltype2 luic lttename1 ltraname lspeed

505! lspeed2 lscale} into L;

506 read all variables {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1

506! traname speed speed2 mgale} into X;

507 use covf84;

508 read all var _num_ into V;

509 dbeta = L*V;

ERROR: (execution) Matrices do not conform to the operation.

operation : * at line 509 column 12

operands : L, V

L 9318 rows 12 cols (numeric)

V 24 rows 24 cols (numeric)

statement : ASSIGN at line 509 column 3

510 ld = J(100, 1, 0);

511 do i = 1 to 100;

512 ld = L[i,]*t(dbeta[i,]);

513 end;

ERROR: (execution) Matrix has not been set to a value.

operation : [ at line 512 column 24

operands : dbeta, i,

dbeta 0 row 0 col (type ?, size 0)

i 1 row 1 col (numeric)

statement : ASSIGN at line 512 column 3

514 W = X || dbeta ||ld;

ERROR: (execution) Matrices do not conform to the operation.

operation : || at line 514 column 9

operands : X, dbeta, ld

X 9318 rows 14 cols (numeric)

dbeta 0 row 0 col (type ?, size 0)

ld 100 rows 1 col (numeric)

statement : ASSIGN at line 514 column 3

515 create fig84_b var {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1

515! traname speed speed2 mgale dbint dbtype1 dbsupply1 dbperiod dbru dbtype2 dbuic dbttename1

515! dbtraname dbspeed dbspeed2 dbscale ld};

516 append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 516 column 3

517 quit;

NOTE: Exiting IML.

NOTE: The data set WORK.FIG84_B has 0 observations and 27 variables.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE IML used (Total process time):

real time 0.06 seconds

cpu time 0.06 seconds

Re: Scaling for score residual

Rick_SAS — Wed, 12 Dec 2012 13:25:52 GMT

The first error message says that the operate

dbeta = L*V;

failed because the two matrices are not the right size for matrix multiplication. The rest of the error message gives details:

L 9318 rows 12 cols (numeric)

V 24 rows 24 cols (numeric)

In order for L*V to make sense, the number of columns of L (currently 12) needs to be the same as the number of rows of V (currently 24).

I don't know what V stands for, but I'm going to guess it is a matrix of values (covariances? Parameter estimates?) that are related to the variables that you read into L. To fix this, you need to use the subset of entries of V that correspond to the variables in L.

To see what you are reading into V, try this:

read all var _num_ into V[colnames=VNames];

print VNames;

That prints the names of the columns of V. You can then choose only the 12x12 submatrix that corresponds to the variables in L.

Re: Scaling for score residual

Tcook — Wed, 12 Dec 2012 14:01:09 GMT

Hello Rick,

I did look up the variable in V and they are 24 due the fact that there are categorical levels which where split into a different column, unlike in the L file where they are not.

Should I create dummy variable for the original data instead of invoking the "class statement" in proc lifereg there by creating 24 columns in L as well or is there a way I can specify that some of the variable contains in L are categorical so that SAS can split each into a new col just like in V.

Thanks in anticipation of your response.

NB: We I did the print V, I only saw 1 row and 24 col without any datum, is this a problem or was that the info that it needed to display?

Re: Scaling for score residual

Rick_SAS — Wed, 12 Dec 2012 14:10:22 GMT

LIFEREG uses a GLM parameterization of classification effects. Therefore you can use PROC GLMMOD and the OUTDESIGN= option to create the dummy variables. Then run LIFEREG on the dummy variables (without any CLASS stmt). That should give you an L matrix that has 24 columns.

Re: Scaling for score residual

Tcook — Wed, 12 Dec 2012 16:38:30 GMT

@ Rick, I did as you suggested, but been having the error message below. Been on it for quite some time but no success. I would appreciate your help with it.

Thanks

proc iml;

NOTE: IML Ready

1183 use fig84_a ;

1184 read all variables {lint ltype11 lsupply11 lsupply12 lperiod1 lperiod2 lperiod3 lperiod4

1184! lru1 ltype21 luic1

1185 luic2 luic3 luic4 luic5 luic6 luic7 luic8 lttename11 ltraname1 ltraname2 lspeed lspeed2

1185! lscale} into L;

1186 read all variables {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1186! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1187 uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1187! speed2 mgale} into X;

1188 use covf84;

1189 read all var _num_ into V;

1190 H = I(24);

1191 H[24,24]=-1/σ

1192 V1 = H*V*H;

1193 dbeta = L*V;

1194 ld = J(7772, 1, 0);

1195 do i = 1 to 77720;

1196 ld = L[i,]*t(dbeta[i,]);

1197 end;

ERROR: (execution) Invalid subscript or subscript out of range.

operation : [ at line 1196 column 12

operands : L, i,

L 7772 rows 24 cols (numeric)

i 1 row 1 col (numeric)

7773

statement : ASSIGN at line 1196 column 3

1198 W = X || dbeta ||ld;

1199 create fig84_b var {pie_id status time1_cum type11 supply11 supply12 period1 period2

1199! period3 period4 ru1 type21 uic1

1200 uic2 uic3 uic4 uic5 uic6 uic7 uic8 ttename11 traname1 traname2 speed speed2 mgale dbint

1200! dbtype11 dbsupply11 dbsupply12 dbperiod1 dbperiod2 dbperiod3 dbperiod4 dbru1 dbtype21 dbuic1

1201 dbuic2 dbuic3 dbuic4 dbuic5 dbuic6 dbuic7 dbuic8 dbttename11 dbtraname1 dbtraname2 dbspeed

1201! dbspeed2 dbscale ld};

1202 append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 1202 column 3

1203 quit;

Re: Scaling for score residual

Rick_SAS — Wed, 12 Dec 2012 16:48:51 GMT

your DO loop is too long:

do i = 1 to 77720; /* should be nrow(L) */

You can get rid of the loop entirely (and improve efficiency) by using matrix multiplication:

Id = L*dBeta`;

Re: Scaling for score residual

Tcook — Wed, 12 Dec 2012 17:04:27 GMT

tried it but still no headway

1266 proc iml;

NOTE: IML Ready

1267 use fig84_a ;

1268 read all variables {lint ltype11 lsupply11 lsupply12 lperiod1 lperiod2 lperiod3 lperiod4

1268! lru1 ltype21 luic1

1269 luic2 luic3 luic4 luic5 luic6 luic7 luic8 lttename11 ltraname1 ltraname2 lspeed lspeed2

1269! lscale} into L;

1270 read all variables {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1270! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1271 uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1271! speed2 mgale} into X;

1272 use covf84;

1273 read all var _num_ into V;

1274 H = I(24);

1275 H[24,24]=-1/σ

1276 V1 = H*V*H;

1277 dbeta = L*V;

1278 ld = L*dBeta;

ERROR: (execution) Matrices do not conform to the operation.

operation : * at line 1278 column 9

operands : L, dBeta

L 7772 rows 24 cols (numeric)

dbeta 7772 rows 24 cols (numeric)

statement : ASSIGN at line 1278 column 3

1279 end;

ERROR: END does not occur within DO group at line=1279 col=3.

1280 W = X || dbeta ||ld;

1281 create fig84_b var {pie_id status time1_cum type11 supply11 supply12 period1 period2

1281! period3 period4 ru1 type21 uic1

1282 uic2 uic3 uic4 uic5 uic6 uic7 uic8 ttename11 traname1 traname2 speed speed2 mgale dbint

1282! dbtype11 dbsupply11 dbsupply12 dbperiod1 dbperiod2 dbperiod3 dbperiod4 dbru1 dbtype21 dbuic1

1283 dbuic2 dbuic3 dbuic4 dbuic5 dbuic6 dbuic7 dbuic8 dbttename11 dbtraname1 dbtraname2 dbspeed

1283! dbspeed2 dbscale ld};

1284 append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 1284 column 2

1285 quit;

NOTE: Exiting IML.

NOTE: The data set WORK.FIG84_B has 0 observations and 51 variables.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE IML used (Total process time):

real time 0.71 seconds

cpu time 0.15 seconds

Re: Scaling for score residual

Rick_SAS — Wed, 12 Dec 2012 18:10:19 GMT

You didn't transpose dBeta. Copy my code EXACTLY and note that there is a transpose operator before the semicolon. The transpose operator (at least on US keyboards) is next to the '1' key and above the TAB key.

You seem to be having trouble reading the error messages that tell you why your program is failing. Read this article on How to interpret SAS/IML error messages - The DO Loop

Re: Scaling for score residual

Tcook — Wed, 12 Dec 2012 20:47:40 GMT

Thanks Rick for your patience and time.

I still trying to find my way round SAS.

I looked at the link you suggested, I also did run the code but got two new errors, one has to do with storage space and the order with the dimension of W.

Try trouble shooting the dimension of W as well as the space issue but wasn't able.

I supposed I may have to run it on a different system, but if I may ask based on experience do you think the first error may be responsible for the second? If not how do I resolve the second, since, it appears the first is a system problem. Which I guess does not border on SAS programming skill.

Below are the messages

1482 proc iml;

NOTE: IML Ready

1483 use fig84_a ;

1484 read all variables {lint ltype11 lsupply11 lsupply12 lperiod1 lperiod2 lperiod3 lperiod4

1484! lru1 ltype21 luic1

1485 luic2 luic3 luic4 luic5 luic6 luic7 luic8 lttename11 ltraname1 ltraname2 lspeed lspeed2

1485! lscale} into L;

1486 read all variables {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1486! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1487 uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1487! speed2 mgale} into X;

1488 use covf84;

1489 read all var _num_ into V;

1490 dbeta = L*V;

1491 Id = L*dBeta`;

ERROR: (execution) Unable to allocate sufficient memory. At least 483231904 more bytes required.

operation : *` at line 1491 column 9

operands : L, dBeta

L 7772 rows 24 cols (numeric)

dbeta 7772 rows 24 cols (numeric)

statement : ASSIGN at line 1491 column 3

1492 W = X || dbeta ||ld;

1493 create fig84_b var {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1493! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1494 uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1494! speed2 mgale dbint dbtype11 dbtype12 dbsupply11 dbsupply12 dbsupply13 dbperiod1 dbperiod2

1494! dbperiod3 dbperiod4 dbperiod5 dbru1 dbru2 dbtype21 dbtype22 dbuic1

1495 dbuic2 dbuic3 dbuic4 dbuic5 dbuic6 dbuic7 dbuic8 dbuic9 dbttename11 dbttename12 dbtraname1

1495! dbtraname2 dbtraname3 dbspeed dbspeed2 dbscale ld};

1496 append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 1496 column 3

1497 quit;

NOTE: Exiting IML.

NOTE: The data set WORK.FIG84_B has 0 observations and 67 variables.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE IML used (Total process time):

real time 1.03 seconds

cpu time 0.10 seconds

Re: Scaling for score residual

Rick_SAS — Wed, 12 Dec 2012 21:05:28 GMT

I don't understand why you are running out of memory. Your matrices are not very large. Do you have a really old computer without much RAM?

As far as I can tell, your program should use about as much memory as the following:

proc iml;

L = j(7720,24);

X = j(7720,14);

V = j(24,24);

dbeta = L*V;

Id = L*dbeta`;

Try running that code and see if you still get an error.

To reclaim memory, you can close the two open data sets:

close fig84_a;

close covf_84;

Here's more ways to save memory. You don't use X at all, so don't even read it. Just write dBeta and Id to separate data sets (of the same names) and then use a DATA step to merge:

data Fig84_b;

merge Fig84_a dBeta Id;

run;

Re: Scaling for score residual

Tcook — Wed, 12 Dec 2012 22:16:03 GMT

yeah, my RAM size is small. But for now my major problem is the appending W, the last error. I will fix the one with storage by using a different system.

1602 use covf84;

1603 read all var _num_ into V;

1604 L = j(7720,24);

1605 X = j(7720,14);

1606 V = j(24,24);

1607 dbeta = L*V;

1608 Id = L*dbeta`;

ERROR: (execution) Unable to allocate sufficient memory. At least 476787232 more bytes required.

operation : *` at line 1608 column 7

operands : L, dbeta

L 7720 rows 24 cols (numeric)

dbeta 7720 rows 24 cols (numeric)

statement : ASSIGN at line 1608 column 1

1609 W = X || dbeta ||ld;

1610 create fig84_b var {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1

1610! traname speed speed2 mgale dbint dbtype1 dbsupply1 dbperiod dbru dbtype2 dbuic dbttename1

1610! dbtraname dbspeed dbspeed2 dbscale ld};

1611 append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

Re: Scaling for score residual

Rick_SAS — Thu, 13 Dec 2012 13:22:25 GMT

Two comments:

1) The error about W is because of the first error. The ID variable is never assigned because of the first error, so W=X || dbeta || ID

becomes W=X || dbeta, which has a different number of columns than the number of variables that you are specifying.

2) I was wrong about your definition of the ID variable. I thought you were doing a matrix multiply, but when I look back at your original program I see now that you are doing a series of inner products that result in a vector, not a matrix. I apologize for the confusion. You should restore the original idea:

ld = J(nrow(L), 1, 0);

do i = 1 to nrow(L);

ld = L[i,]*t(dbeta[i,]);

end;

In your original code, you used 100 instead of nrow(L). Perhaps that was intentional? I don't know how you are using the ID variable, so do what is correct for your application.

Perhaps by fixing the ID variable (which I broke!) your program will work. So sorry.

Re: Scaling for score residual

1zmm — Thu, 13 Dec 2012 13:48:17 GMT

Remember the rules for matrix multiplication. Not only does the number of rows in the second matrix have to equal the number of columns in the first matrix, but the number of rows in the resulting "product" matrix, Id, equals the number of rows in the first matrix, and the number of columns in this "product" matrix equals the number of columns in the second matrix. In your example, the first matrix, L, has 7,720 rows and 24 columns; the second matrix, dbeta, also has 7,720 rows and 24 columns. When dbeta is transposed, however, dbeta` has 24 rows and 7,720 columns. When you multiply L times the transposed dbeta (dbeta'), the resulting product matrix, Id=L*dbeta`, has 7,720 rows and 7,720 columns, which is too large for your computer to handle and perhaps too large for you to interpret (about 60,000,000 entries).

More plausibly, the size of your final product matrix would be 24*24 = 576 entries. To obtain a matrix of this size, you would have to transpose the original first matrix, L, to L` with dimensions of 24 rows and 7,720 columns, and multiply this transposed matrix by the original second matrix, dbeta, with 7,720 rows and 24 columns. Now, the new "product" matrix, Id2=L`*dbeta, has dimensions of 24 rows and 24 columns, which your computer should have no problem handling.

P.S.: I put the term, "product", in the phrase, "product" matrix, in double quotes above to distinguish the matrix that results from the multiplication of two conformable matrices (as here) from the matrix known as the direct (or Kroenecker) product matrix.

Re: Scaling for score residual

Tcook — Wed, 19 Dec 2012 12:27:43 GMT

Hello Rick/1Zmm;

I tried both methods and resolved the problem with the "memory" warning but I got a different error message:

ERROR: Number of columns in W does not match with the number of variables in the data set.

been trying to resolve but was unable.

First how can I call up the columns of the matrix W. so as to see what exactly the columns are, this will help me to know the columns that shouldn't be. Unless there is another way round it that you could suggest?

Thanks.

Re: Scaling for score residual

Rick_SAS — Wed, 19 Dec 2012 15:01:32 GMT

You can use the NCOL and NROW functions to determine the number of columns and rows, respectively of a matrix.

I also suggest that you download the free chapter "Getting Started with the SAS/IML Matrix Programming Language" from my book Statistical Programming with SAS/IML Software. It is available at SAS Press - Rick Wicklin Author Page

Re: Scaling for score residual

Tcook — Wed, 19 Dec 2012 20:40:17 GMT

Hello Rick/1Zmm,

Thanks a lot for your help.