topic Re: How to filter phrases and regular expressions? in SAS Data Science

How to filter phrases and regular expressions?

stepthom — Sun, 17 Jan 2016 20:26:09 GMT

In SAS Enterprise Miner Workstation 13.2, I'm using some Text Mining nodes to build Text Topics.

However, I noticed lots of phrases and tokens tha I would like filtered out of the data before analysis. Examples include html tags such as "<p>", and boilerplate text such as "This description was written by the Martin Group." I tried adding these things to the list of stop words, but that didn't seem to help: the terms still appeared in the created topics.

Is there a way to filter out multi-word phrases? And is there a way to filter out regular expressions, such as "This description was written by .*"?

Re: How to filter phrases and regular expressions?

PGStats — Sun, 17 Jan 2016 21:00:41 GMT

Regular expression matching is very flexible. There is almost certainly a way to do what you describe. But we need something more concrete to suggest good examples. Please give us a list of phrases that you would want to check and what you would expect as a result.

Re: How to filter phrases and regular expressions?

stepthom — Sun, 17 Jan 2016 22:15:40 GMT

Hi PG Stats,

I think I can handle the construction of the regular expression, that's not a problem. My question was trying to ask, where do I put them? (Which node, which field?) I couldn't find it.

thanks!

Re: How to filter phrases and regular expressions?

M_Maldonado — Sun, 17 Jan 2016 22:41:03 GMT

Half the people will recommend doing this transformations before importing data into EM, half the people will recommend doing it in EM.

If I was to add it on EM, I would do it on a transform node (use the SAS code ellipsis!), hptransform node, or in a SAS code node.

good luck!