01-31-2012 04:04 PM
I'm trying to use Proc Varclus and I'm getting a lot of
ERROR: All observations in data set WORK.PROTEIN have missing values, or the sum of weights or
frequencies is nonpositive.
because of the missing values in my data.
This is my code
proc varclus data=IN outtree=OUT centroid;
Is there a way around it or would proc varclus work fine if I make the missing to 0's. The values are averages of different economical growth rates. So for example average growth rate ot the unemployment rate in the past 5 years, and average GDP growth rate in the past 5 years, etc... Thanks
01-31-2012 05:50 PM
For averages of growth rate you're setting 0, which is assuming a stable system. I think 0 would be misleading, especially as every observation is missing a variable.
I'd double check why the variables are missing and seeing if there was something logically incorrect with my calculations or if there was another way to interpolate these numbers than assume 0.
02-01-2012 08:01 AM
If it is indeed the case that every observation has at least one missing value, you might need to choose another approach. Three ideas:
1) There might be one or two (or 10) variables that contain mostly missing values, whereas the other 999 variables don't contain ANY missings.
If so, you can exclude those "bad" variables and just use the good ones. To count the number of missing values for each variable, use
proc means data=IN NMISS N; var VAR01-Var200; run;
2) If your missing values are scattered at random throughout the variables, you might be able to impute values for the missings by using PROC MI, and then analyze the imputed variables. However, if you have too many missing values (like 60%). I'm not sure I would trust the MI results.
3) Instead of VARCLUS, you can use PROC CORR to look at PAIRWISE correlations between variables. A pairwise approach uses all observations in common between a pair of variables, whereas the LISTWISE approach that you are currently using requires all 200 variables be nonmissinging.There are then various ways to cluster variables based on the pairwise correlations. For example, see
C. Hurley (2004) "Clustering Visualisations of Multidimensional Data", J. Comp. and Graphical Stats, 13(4), pp 788-806
02-01-2012 09:33 AM
Rick, would it also make sense to use PROC EXPAND to intrapolate the missing values (whether they're growth rates or the actual values) and then VARCLUS or PROC SIMILARITY woudl work? Since I do have a Date stamp ..
02-01-2012 09:41 AM
It might, if the time series aren't too wild and noisy.
You need to do (1) because if there are variables that contain only a few nonmissing observations, those should be dropped from the analysis. Using PROC EXPAND (or any interpolation/imputation scheme) on data that has only a few valid points is not a good idea.