i want to perform text analysis on a tweet and i want to know if i still need to do some text preprocessing, such as removing special characters, punctuation, tokenization, and stop words, before importing into model and creating pipe line
I'm not a text analytics expert, but common sense suggests that you have asked a question that cannot be accurately answered without you running some tests yourself.
We don't know what your data looks like or how well it might be scanned without it being cleaned up first. You should try running some test text analytics and see how well it identifies your words or phrases of interest. You might get lucky and find that it is accurate without preprocessing but given the nature of tweets I doubt this.
I'm not a text analytics expert, but common sense suggests that you have asked a question that cannot be accurately answered without you running some tests yourself.
We don't know what your data looks like or how well it might be scanned without it being cleaned up first. You should try running some test text analytics and see how well it identifies your words or phrases of interest. You might get lucky and find that it is accurate without preprocessing but given the nature of tweets I doubt this.
Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.
Register today!Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.
Find more tutorials on the SAS Users YouTube channel.