The effect of one variable's level cannot be estimated if...

bncoxuk — Wed, 17 Aug 2011 23:37:15 GMT

I just did an experiment. It showed that one level of a variable cannot be estimated if this level has a perfect correlation with a level of another variables. Let's see the data below:

data work.data;

infile datalines;

input y x1 x2 @@;

datalines;

0 1 2 0 1 2 0 1 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1

0 2 2 0 2 1 0 2 2 0 2 2 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 2

0 2 2 0 2 2 0 2 2 0 3 3 0 3 3 0 3 2 0 3 2 0 3 1 0 3 1 0 3 1

1 1 1 1 1 1 1 3 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 1 1 2 1 1 2 1

1 3 3 1 3 2 1 3 3 1 3 2 1 3 2 1 3 3 0 4 4 0 4 4 1 4 4 1 4 4

0 3 2 0 1 2 0 1 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1

0 2 2 0 2 1 0 2 2 0 2 2 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 2

0 2 2 0 2 2 0 2 2 0 3 3 0 3 3 0 3 2 0 3 2 0 3 1 0 3 1 0 3 1

1 3 1 1 3 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 1 1 2 1 1 2 1

1 3 3 1 3 2 1 3 3 1 3 2 1 3 1 1 3 3 0 4 4 0 4 4 1 4 4 1 4 4

run;

You can see whenever x1=4, then x2 also eqals to 4.

Let's run some logistic regression:

proc logistic data=work.data;

class x1 x2 /param=glm;

model y=x1 x2 /aggregate scale=none;

run;

Results:

Standard Wald

Parameter DF Estimate Error Chi-Square Pr > ChiSq

Intercept 1 4.82E-15 0.7071 0.0000 1.0000

x1 1 1 -1.6953 1.1763 2.0770 0.1495

x1 2 1 1.4020 1.1598 1.4612 0.2267

x1 3 1 -0.4055 0.9574 0.1793 0.6719

x1 4 0 0 . . .

x2 1 1 -0.0799 0.8535 0.0088 0.9254

x2 2 1 1.3451 0.8710 2.3847 0.1225

x2 3 0 0 . . .

x2 4 0 0 . . .

You can see the results: x2=4 is the reference group, so DF=0. But x2=3 also has DF=0. Problem! This is because x2=4 is 100% correlated with x1=4. You can say in this case x2=3 acts as the reference group, and the effect of x2=4 has been explained by the other variable x1's x1=4.

Let's run a linear regression model using PROC GLM:

proc glm data=work.data;

class x1 x2;

model y=x1 x2/solution ss3;

run;

quit;

The results are the same as below. There is no estimate for x2=3.

Standard

Parameter Estimate Error t Value Pr > |t|

Intercept 0.5000000000 B 0.14821413 3.37 0.0010

x1 1 0.3879750199 B 0.23428608 1.66 0.1005

x1 2 -.2299495084 B 0.22891720 -1.00 0.3173

x1 3 0.1000000000 B 0.19885012 0.50 0.6160

x1 4 0.0000000000 B . . .

x2 1 -.0381211799 B 0.16953676 -0.22 0.8225

x2 2 -.2618788201 B 0.16953676 -1.54 0.1252

x2 3 0.0000000000 B . . .

x2 4 0.0000000000 B . . .

If you change the data so that x1=4 and x2=4 are not perfectly related, then the problem disappeared.

I think the explanation can be that 'the effects of the levels can only be estimated unless they have unique contribution, i.e. they are not perfectly explained by other variables'.

Any better thoughts, statistically?

The effect of one variable's level cannot be estimated if...

Doc_Duke — Thu, 18 Aug 2011 03:17:37 GMT

Correct. You can see it more clearly with continuous variables. If you have a model with, say, systolic blood pressure, diastolic blood pressure and pulse pressure (= (systolic - diastolic) ), then any two variables carry the same infomation as the third. You can actually write it out

y = int + b1*systolic + b2*diastolic + b3*pulse

= int + b1*systolic + b2*diastolic + b3*(systolic - diastolic)

= int + b1*systolic + b2*diastolic + b3*systolic - b3*diastolic

=int + (b1+b3)*systolic + (b2-b3)*diastolic

The effect of one variable's level cannot be estimated if...

bncoxuk — Thu, 18 Aug 2011 07:45:20 GMT

Hi Duke, your example gives pulse pressue as being perfectly explained by systolic and diastolic: pulse pressure (= (systolic - diastolic) )

But in my data, the relationship is not that tight: not the whole variable is perfectly explained, but only one of its 4 levels.

I assume the principle is the same.

The effect of one variable's level cannot be estimated if...

Reeza — Fri, 19 Aug 2011 16:33:48 GMT

For regression with categorical variables essentially dummy variables get created for n-1 levels with the nth level being all 0.

So if one level correlates perfectly with another level in a different categorical variable, the two are linear combinations of each other, same as Doc@Duke example.

Typically you can change the way you categorize the variables to help solve this one, but in your case with all missing for a particular population it might be tricky. You also have systematic missing data, which is a problem in of itself.

topic The effect of one variable's level cannot be estimated if... in SAS Procedures

The effect of one variable's level cannot be estimated if...

The effect of one variable's level cannot be estimated if...

The effect of one variable's level cannot be estimated if...

The effect of one variable's level cannot be estimated if...