topic missing values with proc varclus in Statistical Procedures

missing values with proc varclus

podarum — Tue, 31 Jan 2012 21:04:49 GMT

Hi,

I'm trying to use Proc Varclus and I'm getting a lot of

ERROR: All observations in data set WORK.PROTEIN have missing values, or the sum of weights or

frequencies is nonpositive.

because of the missing values in my data.

This is my code

proc varclus data=IN outtree=OUT centroid;

var VAR01-VAR200;

run;

Is there a way around it or would proc varclus work fine if I make the missing to 0's. The values are averages of different economical growth rates. So for example average growth rate ot the unemployment rate in the past 5 years, and average GDP growth rate in the past 5 years, etc... Thanks

missing values with proc varclus

Reeza — Tue, 31 Jan 2012 22:50:20 GMT

For averages of growth rate you're setting 0, which is assuming a stable system. I think 0 would be misleading, especially as every observation is missing a variable.

I'd double check why the variables are missing and seeing if there was something logically incorrect with my calculations or if there was another way to interpolate these numbers than assume 0.

missing values with proc varclus

Rick_SAS — Wed, 01 Feb 2012 13:01:35 GMT

If it is indeed the case that every observation has at least one missing value, you might need to choose another approach. Three ideas:

1) There might be one or two (or 10) variables that contain mostly missing values, whereas the other 999 variables don't contain ANY missings.

If so, you can exclude those "bad" variables and just use the good ones. To count the number of missing values for each variable, use

proc means data=IN NMISS N; var VAR01-Var200; run;

2) If your missing values are scattered at random throughout the variables, you might be able to impute values for the missings by using PROC MI, and then analyze the imputed variables. However, if you have too many missing values (like 60%). I'm not sure I would trust the MI results.

3) Instead of VARCLUS, you can use PROC CORR to look at PAIRWISE correlations between variables. A pairwise approach uses all observations in common between a pair of variables, whereas the LISTWISE approach that you are currently using requires all 200 variables be nonmissinging.There are then various ways to cluster variables based on the pairwise correlations. For example, see

C. Hurley (2004) "Clustering Visualisations of Multidimensional Data", J. Comp. and Graphical Stats, 13(4), pp 788-806

missing values with proc varclus

podarum — Wed, 01 Feb 2012 14:11:19 GMT

Thanks Rick, I'll try 2 and 3 ...

missing values with proc varclus

podarum — Wed, 01 Feb 2012 14:33:20 GMT

Rick, would it also make sense to use PROC EXPAND to intrapolate the missing values (whether they're growth rates or the actual values) and then VARCLUS or PROC SIMILARITY woudl work? Since I do have a Date stamp ..

missing values with proc varclus

Rick_SAS — Wed, 01 Feb 2012 14:41:52 GMT

It might, if the time series aren't too wild and noisy.

You need to do (1) because if there are variables that contain only a few nonmissing observations, those should be dropped from the analysis. Using PROC EXPAND (or any interpolation/imputation scheme) on data that has only a few valid points is not a good idea.