i want to perform text analysis on a tweet and i want to know if i still need to do some text preprocessing, such as removing special characters, punctuation, tokenization, and stop words, before importing into model and creating pipe line
I'm not a text analytics expert, but common sense suggests that you have asked a question that cannot be accurately answered without you running some tests yourself.
We don't know what your data looks like or how well it might be scanned without it being cleaned up first. You should try running some test text analytics and see how well it identifies your words or phrases of interest. You might get lucky and find that it is accurate without preprocessing but given the nature of tweets I doubt this.
I'm not a text analytics expert, but common sense suggests that you have asked a question that cannot be accurately answered without you running some tests yourself.
We don't know what your data looks like or how well it might be scanned without it being cleaned up first. You should try running some test text analytics and see how well it identifies your words or phrases of interest. You might get lucky and find that it is accurate without preprocessing but given the nature of tweets I doubt this.
Join us for SAS Innovate 2025, our biggest and most exciting global event of the year, in Orlando, FL, from May 6-9. Sign up by March 14 for just $795.
Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.
Find more tutorials on the SAS Users YouTube channel.