topic Re: Need Advice on Handling High-Dimensional Data in Data Science Project in SAS Data Science

Need Advice on Handling High-Dimensional Data in Data Science Project

tress — Sun, 28 Apr 2024 11:57:46 GMT

Hey everyone,

I’m relatively new to data science and currently working on a project that involves a dataset with over 60 columns. Many of these columns are categorical, with more than 100 unique values each.

My issue arises when I try to apply one-hot encoding to these categorical columns. It seems like I’m running into the curse of dimensionality problem, and I’m not quite sure how to proceed from here.

I’d really appreciate some advice or guidance on how to effectively handle high-dimensional data in this context. Are there alternative encoding techniques I should consider? Or perhaps there are preprocessing steps I’m overlooking?

Any insights or tips would be immensely helpful.

Thanks in advance!

Re: Need Advice on Handling High-Dimensional Data in Data Science Project

PaigeMiller — Sun, 28 Apr 2024 12:44:58 GMT

Can you describe the project you are working on? Can you tell us what the desired outcome(s) is of the analysis of this data? When you say you "need advice on handling", well there are a gazillion ways to "handle" such data and unless we know what desired outcome(s) are, we can't really give you good advice.

Re: Need Advice on Handling High-Dimensional Data in Data Science Project

FreelanceReinh — Sun, 28 Apr 2024 13:44:35 GMT

Also, have you already tried some of the suggestions from the Reddit thread you copied your post from, e.g., to "aggregate the 100+ categories into a smaller set of categories"?

Re: Need Advice on Handling High-Dimensional Data in Data Science Project

Patrick — Mon, 29 Apr 2024 01:55:33 GMT

Please provide some SAS related response here so we can verify that this is not once more just spam.

Re: Need Advice on Handling High-Dimensional Data in Data Science Project

sbxkoenk — Mon, 29 Apr 2024 13:10:12 GMT

@tress wrote:

Many of these columns are categorical, with more than 100 unique values each.

My issue arises when I try to apply one-hot encoding to these categorical columns. It seems like I’m running into the curse of dimensionality problem, and I’m not quite sure how to proceed from here.

You can do level clustering (taking the target variable into account) on your class (categorical) inputs.
Something simpler (and unsupervised) is the Group Rare Levels Transformation (for class variables only).
This method combines the rare levels (if any) into a separate group, _OTHER_. Rare levels are the variable values that occur less than the specified cutoff value. I would start with a cutoff value of 5% (0.05).

BR, Koen