Scaling for score residual

[Tcook]

Hi all,

I am trying to generate a residual plot for a weibull survival model based on the code used in "SAS Textbook Examples
Applied Survival Analysis by Hosmer, Lemeshow and May Chapter 8: Parametric Regression Methods" but I kept getting an error message about the matrices and I have tried all I could be to no avail. The fig84_a file in my own case has 9318 entries and 12 varaiables.

I would greatly appreciate if someone can take look at it and point out what exactly I need to do to fix it.

Below is my code and and error message received.

/*********code******************************/

proc iml;

  use fig84_a ;

  read all variables {lint ltype1 lsupply1 lperiod lru ltype2 luic lttename1 ltraname lspeed lspeed2 lscale} into L;

  read all variables {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1 traname speed speed2 mgale} into X;

  use covf84;

  read all  var _num_ into V;

  dbeta = L*V;

  ld = J(100, 1, 0);

  do i = 1 to 100;

  ld = L[i,]*t(dbeta[i,]);

  end;

  W = X || dbeta ||ld;

  create fig84_b var {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1 traname speed speed2 mgale dbint dbtype1 dbsupply1 dbperiod dbru dbtype2 dbuic dbttename1 dbtraname dbspeed dbspeed2 dbscale ld};

  append from W;

quit;

/***********ERROR MESSAGE************************

503  proc iml;

NOTE: IML Ready

504    use fig84_a ;

505    read all variables {lint ltype1 lsupply1 lperiod lru ltype2 luic lttename1 ltraname lspeed

505! lspeed2 lscale} into L;

506    read all variables {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1

506! traname speed speed2 mgale} into X;

507    use covf84;

508    read all  var _num_ into V;

509    dbeta = L*V;

ERROR: (execution) Matrices do not conform to the operation.

operation : * at line 509 column 12

operands  : L, V

L   9318 rows     12 cols    (numeric)

V     24 rows     24 cols    (numeric)

statement : ASSIGN at line 509 column 3

510    ld = J(100, 1, 0);

511    do i = 1 to 100;

512    ld = L[i,]*t(dbeta[i,]);

513    end;

ERROR: (execution) Matrix has not been set to a value.

operation : [ at line 512 column 24

operands  : dbeta, i,

dbeta      0 row       0 col     (type ?, size 0)

i      1 row       1 col     (numeric)

         1

statement : ASSIGN at line 512 column 3

514    W = X || dbeta ||ld;

ERROR: (execution) Matrices do not conform to the operation.

operation : || at line 514 column 9

operands  : X, dbeta, ld

X   9318 rows     14 cols    (numeric)

dbeta      0 row       0 col     (type ?, size 0)

ld    100 rows      1 col     (numeric)

statement : ASSIGN at line 514 column 3

515    create fig84_b var {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1

515! traname speed speed2 mgale dbint dbtype1 dbsupply1 dbperiod dbru dbtype2 dbuic dbttename1

515! dbtraname dbspeed dbspeed2 dbscale ld};

516    append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 516 column 3

517  quit;

NOTE: Exiting IML.

NOTE: The data set WORK.FIG84_B has 0 observations and 27 variables.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE IML used (Total process time):

      real time           0.06 seconds

      cpu time            0.06 seconds

[Rick_SAS]

The first error message says that the operate

dbeta = L*V;

failed because the two matrices are not the right size for matrix multiplication. The rest of the error message gives details:

L   9318 rows     12 cols    (numeric)

V     24 rows     24 cols    (numeric)

In order for L*V to make sense, the number of columns of L (currently 12) needs to be the same as the number of rows of V (currently 24).

I don't know what V stands for, but I'm going to guess it is a matrix of values (covariances? Parameter estimates?) that are related to the variables that you read into L. To fix this, you need to use the subset of entries of V that correspond to the variables in L.

To see what you are reading into V, try this:

read all  var _num_ into V[colnames=VNames];

print VNames;

That prints the names of the columns of V.  You can then choose only the 12x12 submatrix that corresponds to the variables in L.

[Tcook]

Hello Rick,

I did look up the variable in V and they are 24 due the fact that there are categorical levels which where split into a different column, unlike in the L file where they are not.

Should I create dummy variable for the original data instead of invoking the "class statement" in proc lifereg there by creating 24 columns in L as well or is there a way I can specify that some of the variable contains in L are categorical so that SAS can split each into a new col just like in V.

Thanks in anticipation of your response.

NB: We I did the print V, I only saw 1 row and 24 col without any datum, is this a problem or was that the info that it needed to display?

[Rick_SAS]

LIFEREG uses a GLM parameterization of classification effects. Therefore you can use PROC GLMMOD and the OUTDESIGN= option to create the dummy variables. Then run LIFEREG on the dummy variables (without any CLASS stmt).  That should give you an L matrix that has 24 columns.

[Tcook]

@ Rick, I did as you suggested, but been having the error message below. Been on it for quite some time but no success.  I would appreciate your help with it.

Thanks

proc iml;

NOTE: IML Ready

1183    use fig84_a ;

1184    read all variables {lint ltype11 lsupply11 lsupply12 lperiod1 lperiod2 lperiod3 lperiod4

1184! lru1 ltype21 luic1

1185  luic2 luic3 luic4 luic5 luic6 luic7 luic8 lttename11  ltraname1 ltraname2 lspeed lspeed2

1185! lscale} into L;

1186    read all variables {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1186! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1187  uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1187! speed2 mgale} into X;

1188    use covf84;

1189      read all  var _num_ into V;

1190      H = I(24);

1191    H[24,24]=-1/σ

1192    V1 = H*V*H;

1193    dbeta = L*V;

1194    ld = J(7772, 1, 0);

1195    do i = 1 to 77720;

1196    ld = L[i,]*t(dbeta[i,]);

1197    end;

ERROR: (execution) Invalid subscript or subscript out of range.

operation : [ at line 1196 column 12

operands  : L, i,

L   7772 rows     24 cols    (numeric)

i      1 row       1 col     (numeric)

      7773

statement : ASSIGN at line 1196 column 3

1198    W = X || dbeta ||ld;

1199    create fig84_b var {pie_id status time1_cum type11 supply11 supply12 period1 period2

1199! period3 period4 ru1 type21 uic1

1200  uic2 uic3 uic4 uic5 uic6 uic7 uic8 ttename11 traname1 traname2 speed speed2 mgale dbint

1200! dbtype11 dbsupply11 dbsupply12 dbperiod1 dbperiod2 dbperiod3 dbperiod4 dbru1 dbtype21 dbuic1

1201  dbuic2 dbuic3 dbuic4 dbuic5 dbuic6 dbuic7 dbuic8 dbttename11 dbtraname1 dbtraname2 dbspeed

1201! dbspeed2 dbscale ld};

1202    append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 1202 column 3

1203  quit;

[Rick_SAS]

your DO loop is too long:

do i = 1 to 77720; /* should be nrow(L) */

You can get rid of the loop entirely (and improve efficiency) by using matrix multiplication:

Id = L*dBeta`;

[Tcook]

tried it but still no headway

1266  proc iml;

NOTE: IML Ready

1267    use fig84_a ;

1268    read all variables {lint ltype11 lsupply11 lsupply12 lperiod1 lperiod2 lperiod3 lperiod4

1268! lru1 ltype21 luic1

1269  luic2 luic3 luic4 luic5 luic6 luic7 luic8 lttename11  ltraname1 ltraname2 lspeed lspeed2

1269! lscale} into L;

1270    read all variables {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1270! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1271  uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1271! speed2 mgale} into X;

1272    use covf84;

1273      read all  var _num_ into V;

1274      H = I(24);

1275    H[24,24]=-1/σ

1276    V1 = H*V*H;

1277    dbeta = L*V;

1278    ld = L*dBeta;

ERROR: (execution) Matrices do not conform to the operation.

operation : * at line 1278 column 9

operands  : L, dBeta

L   7772 rows     24 cols    (numeric)

dbeta   7772 rows     24 cols    (numeric)

statement : ASSIGN at line 1278 column 3

1279    end;

ERROR: END does not occur within DO group at line=1279 col=3.

1280    W = X || dbeta ||ld;

1281    create fig84_b var {pie_id status time1_cum type11 supply11 supply12 period1 period2

1281! period3 period4 ru1 type21 uic1

1282  uic2 uic3 uic4 uic5 uic6 uic7 uic8 ttename11 traname1 traname2 speed speed2 mgale dbint

1282! dbtype11 dbsupply11 dbsupply12 dbperiod1 dbperiod2 dbperiod3 dbperiod4 dbru1 dbtype21 dbuic1

1283  dbuic2 dbuic3 dbuic4 dbuic5 dbuic6 dbuic7 dbuic8 dbttename11 dbtraname1 dbtraname2 dbspeed

1283! dbspeed2 dbscale ld};

1284   append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 1284 column 2

1285  quit;

NOTE: Exiting IML.

NOTE: The data set WORK.FIG84_B has 0 observations and 51 variables.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE IML used (Total process time):

      real time           0.71 seconds

      cpu time            0.15 seconds

[Rick_SAS]

You didn't transpose dBeta. Copy my code EXACTLY and note that there is a transpose operator before the semicolon. The transpose operator (at least on US keyboards) is next to the '1' key and above the TAB key.

You seem to be having trouble reading the error messages that tell you why your program is failing. Read this article on How to interpret SAS/IML error messages - The DO Loop

[Tcook]

Thanks Rick for your patience and time. 

I still trying to find my way round SAS.

I looked at the link you suggested, I also did run the code but got two new errors, one has to do with storage space and the order with the dimension of W.

Try trouble shooting the dimension of W as well as the space issue but wasn't able.

I supposed I may have to run it on a different system, but if I may ask based on experience do you think the first error may be responsible for the second? If not how do I resolve the second, since, it appears the first is a system problem. Which I guess does not border on SAS programming skill.

Below are the messages

1482  proc iml;

NOTE: IML Ready

1483    use fig84_a ;

1484    read all variables {lint ltype11 lsupply11 lsupply12 lperiod1 lperiod2 lperiod3 lperiod4

1484! lru1 ltype21 luic1

1485  luic2 luic3 luic4 luic5 luic6 luic7 luic8 lttename11  ltraname1 ltraname2 lspeed lspeed2

1485! lscale} into L;

1486    read all variables {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1486! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1487  uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1487! speed2 mgale} into X;

1488    use covf84;

1489      read all  var _num_ into V;

1490    dbeta = L*V;

1491    Id = L*dBeta`;

ERROR: (execution) Unable to allocate sufficient memory. At least 483231904 more bytes required.

operation : *` at line 1491 column 9

operands  : L, dBeta

L   7772 rows     24 cols    (numeric)

dbeta   7772 rows     24 cols    (numeric)

statement : ASSIGN at line 1491 column 3

1492    W = X || dbeta ||ld;

1493    create fig84_b var {pie_id status time1_cum type11 type12 supply11 supply12 supply13

1493! period1 period2 period3 period4 period5 ru1 ru2 type21 type22 uic1

1494  uic2 uic3 uic4 uic5 uic6 uic7 uic8 uic9 ttename11 ttename12 traname1 traname2 traname3 speed

1494! speed2 mgale dbint dbtype11 dbtype12 dbsupply11 dbsupply12 dbsupply13 dbperiod1 dbperiod2

1494! dbperiod3 dbperiod4 dbperiod5 dbru1 dbru2 dbtype21 dbtype22 dbuic1

1495  dbuic2 dbuic3 dbuic4 dbuic5 dbuic6 dbuic7 dbuic8 dbuic9 dbttename11 dbttename12 dbtraname1

1495! dbtraname2 dbtraname3 dbspeed dbspeed2 dbscale ld};

1496    append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

statement : APPEND at line 1496 column 3

1497  quit;

NOTE: Exiting IML.

NOTE: The data set WORK.FIG84_B has 0 observations and 67 variables.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE IML used (Total process time):

      real time           1.03 seconds

      cpu time            0.10 seconds

[Rick_SAS]

I don't understand why you are running out of memory. Your matrices are not very large. Do you have a really old computer without much RAM?

As far as I can tell, your program should use about as much memory as the following:

proc iml;

L = j(7720,24);

X = j(7720,14);

V = j(24,24);

dbeta = L*V;

Id = L*dbeta`;

Try running that code and see if you still get an error.

To reclaim memory, you can close the two open data sets:

close  fig84_a;

close covf_84;

Here's more ways to save memory. You don't use X at all, so don't even read it. Just write dBeta and Id to separate data sets (of the same names) and then use a DATA step to merge:

data Fig84_b;

merge Fig84_a dBeta Id;

run;

[Tcook]

yeah, my RAM size is small.  But for now my major problem is the appending W, the last error. I will fix the one with storage by using a different system.

1602    use covf84;

1603      read all  var _num_ into V;

1604  L = j(7720,24);

1605  X = j(7720,14);

1606  V = j(24,24);

1607  dbeta = L*V;

1608  Id = L*dbeta`;

ERROR: (execution) Unable to allocate sufficient memory. At least 476787232 more bytes required.

operation : *` at line 1608 column 7

operands  : L, dbeta

L   7720 rows     24 cols    (numeric)

dbeta   7720 rows     24 cols    (numeric)

statement : ASSIGN at line 1608 column 1

1609  W = X || dbeta ||ld;

1610    create fig84_b var {pie_id status time1_cum type1 supply1 period ru type2 uic ttename1

1610! traname speed speed2 mgale dbint dbtype1 dbsupply1 dbperiod dbru dbtype2 dbuic dbttename1

1610! dbtraname dbspeed dbspeed2 dbscale ld};

1611    append from W;

ERROR: Number of columns in W does not match with the number of variables in the data set.

[Rick_SAS]

Two comments:

1) The error about W is because of the first error. The ID variable is never assigned because of the first error, so W=X || dbeta || ID

becomes  W=X || dbeta, which has a different number of columns than the number of variables that you are specifying.

2) I was wrong about your definition of the ID variable.  I thought you were doing a matrix multiply, but when I look back at your original program I see now that you are doing a series of inner products that result in a vector, not a matrix.  I apologize for the confusion. You should restore the original idea:

  ld = J(nrow(L), 1, 0);

  do i = 1 to nrow(L);

  ld = L[i,]*t(dbeta[i,]);

  end;

In your original code, you used 100 instead of nrow(L).  Perhaps that was intentional? I don't know how you are using the ID variable, so do what is correct for your application.

Perhaps by fixing the ID variable (which I broke!) your program will work.  So sorry.

[Tcook]

Hello Rick/1Zmm;

I tried both methods and resolved the problem with the "memory" warning but I got a different error message:

ERROR: Number of columns in W does not match with the number of variables in the data set.

been trying to resolve but was unable.

First how can I call up the columns of the matrix W. so as to see what exactly the columns are, this will help me to know the columns that shouldn't be. Unless there is another way round it that you could suggest?

Thanks.

[1zmm]

Remember the rules for matrix multiplication.  Not only does the number of rows in the second matrix have to equal the number of columns in the first matrix, but the number of rows in the resulting "product" matrix, Id, equals the number of rows in the first matrix, and the number of columns in this "product" matrix equals the number of columns in the second matrix.  In your example, the first matrix, L, has 7,720 rows and 24 columns; the second matrix, dbeta, also has 7,720 rows and 24 columns.  When dbeta is transposed, however, dbeta` has 24 rows and 7,720 columns.  When you multiply L times the transposed dbeta (dbeta'), the resulting product matrix, Id=L*dbeta`, has 7,720 rows and 7,720 columns, which is too large for your computer to handle and perhaps too large for you to interpret (about 60,000,000 entries).

More plausibly, the size of your final product matrix would be 24*24 = 576 entries.  To obtain a matrix of this size, you would have to transpose the original first matrix, L, to L` with dimensions of 24 rows and 7,720 columns, and multiply this transposed matrix by the original second matrix, dbeta, with 7,720 rows and 24 columns.  Now, the new "product" matrix, Id2=L`*dbeta, has dimensions of 24 rows and 24 columns, which your computer should have no problem handling.

P.S.:  I put the term, "product", in the phrase, "product" matrix, in double quotes above to distinguish the matrix that results from the  multiplication of two conformable matrices (as here) from the matrix known as the direct (or Kroenecker) product matrix.

[Rick_SAS · Accepted solution]

You can use the NCOL and NROW functions to determine the number of columns and rows, respectively of a matrix.

I also suggest that you download the free chapter "Getting Started with the SAS/IML Matrix Programming Language" from my book Statistical Programming with SAS/IML Software. It is available at SAS Press - Rick Wicklin Author Page