Need Advice on Handling High-Dimensional Data in Data Science Project

[tress]

Hey everyone,

I’m relatively new to data science and currently working on a project that involves a dataset with over 60 columns. Many of these columns are categorical, with more than 100 unique values each.

My issue arises when I try to apply one-hot encoding to these categorical columns. It seems like I’m running into the curse of dimensionality problem, and I’m not quite sure how to proceed from here.

I’d really appreciate some advice or guidance on how to effectively handle high-dimensional data in this context. Are there alternative encoding techniques I should consider? Or perhaps there are preprocessing steps I’m overlooking?

Any insights or tips would be immensely helpful.

Thanks in advance!

[PaigeMiller]

Can you describe the project you are working on? Can you tell us what the desired outcome(s) is of the analysis of this data? When you say you "need advice on handling", well there are a gazillion ways to "handle" such data and unless we know what desired outcome(s) are, we can't really give you good advice.

[FreelanceReinh]

Also, have you already tried some of the suggestions from the Reddit thread you copied your post from, e.g., to "aggregate the 100+ categories into a smaller set of categories"?

[Patrick]

Please provide some SAS related response here so we can verify that this is not once more just spam.

[sbxkoenk]

---

@tress wrote:

Many of these columns are categorical, with more than 100 unique values each.

My issue arises when I try to apply one-hot encoding to these categorical columns. It seems like I’m running into the curse of dimensionality problem, and I’m not quite sure how to proceed from here.

• You can do level clustering (taking the target variable into account) on your class (categorical) inputs.
• Something simpler (and unsupervised) is the Group Rare Levels Transformation (for class variables only).
  This method combines the rare levels (if any) into a separate group, _OTHER_. Rare levels are the variable values that occur less than the specified cutoff value. I would start with a cutoff value of 5% (0.05).

BR, Koen