does sas viya Text analytics doesnt require text preprocessing

[Ifunanya]

i want to perform text analysis on a tweet and i want to know if i still need to do some text preprocessing, such as removing special characters, punctuation, tokenization, and stop words, before importing into model and creating pipe line

[SASKiwi · Accepted solution]

I'm not a text analytics expert, but common sense suggests that you have asked a question that cannot be accurately answered without you running some tests yourself.

We don't know what your data looks like or how well it might be scanned without it being cleaned up first. You should try running some test text analytics and see how well it identifies your words or phrases of interest. You might get lucky and find that it is accurate without preprocessing but given the nature of tweets I doubt this.