SAS Data Science

Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Viya (Machine Learning), SAS Visual Text Analytics, with point-and-click interfaces or programming
BookmarkSubscribeRSS Feed
tress
Calcite | Level 5

Hey everyone,

I’m relatively new to data science and currently working on a project that involves a dataset with over 60 columns. Many of these columns are categorical, with more than 100 unique values each.

My issue arises when I try to apply one-hot encoding to these categorical columns. It seems like I’m running into the curse of dimensionality problem, and I’m not quite sure how to proceed from here.

I’d really appreciate some advice or guidance on how to effectively handle high-dimensional data in this context. Are there alternative encoding techniques I should consider? Or perhaps there are preprocessing steps I’m overlooking?

Any insights or tips would be immensely helpful.

Thanks in advance!

4 REPLIES 4
PaigeMiller
Diamond | Level 26

Can you describe the project you are working on? Can you tell us what the desired outcome(s) is of the analysis of this data? When you say you "need advice on handling", well there are a gazillion ways to "handle" such data and unless we know what desired outcome(s) are, we can't really give you good advice.

--
Paige Miller
FreelanceReinh
Jade | Level 19

Also, have you already tried some of the suggestions from the Reddit thread you copied your post from, e.g., to "aggregate the 100+ categories into a smaller set of categories"?

Patrick
Opal | Level 21

Please provide some SAS related response here so we can verify that this is not once more just spam.

sbxkoenk
SAS Super FREQ

@tress wrote:

Many of these columns are categorical, with more than 100 unique values each.

My issue arises when I try to apply one-hot encoding to these categorical columns. It seems like I’m running into the curse of dimensionality problem, and I’m not quite sure how to proceed from here.

  • You can do level clustering (taking the target variable into account) on your class (categorical) inputs.
  • Something simpler (and unsupervised) is the Group Rare Levels Transformation (for class variables only).
    This method combines the rare levels (if any) into a separate group, _OTHER_. Rare levels are the variable values that occur less than the specified cutoff value. I would start with a cutoff value of 5% (0.05).

BR, Koen

sas-innovate-white.png

Our biggest data and AI event of the year.

Don’t miss the livestream kicking off May 7. It’s free. It’s easy. And it’s the best seat in the house.

Join us virtually with our complimentary SAS Innovate Digital Pass. Watch live or on-demand in multiple languages, with translations available to help you get the most out of every session.

 

Register now!

How to choose a machine learning algorithm

Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 4 replies
  • 1648 views
  • 4 likes
  • 5 in conversation