topic Re: Question about exported data from text parsing in SAS Enterprise Miner in SAS Data Science

Question about exported data from text parsing in SAS Enterprise Miner

Jonathanzz — Wed, 17 Apr 2019 07:56:42 GMT

I am trying to use text parsing to keep only useful words in the dataset using my specified start list. However, there is no any change in the text parsing exported dataset. The result shows that only staff, restaurant, swimming pool and spa with "Y" in keep column whereas all other words are "N", however, the exported dataset does not change anything at all.

Original dataset:

review_ID review

0001 I am satisfied with the staff and also the restaurant

0002 The swimming pool and spa are amazing

...

Start list:

staff

restaurant

swimming pool

spa

Expected exported dataset from text parsing:

review_ID review

0001 staff, restaurant

0002 swimming pool, spa

...

Actual exported dataset from text parsing: (not what I want)

review_ID review

0001 I am satisfied with the staff and also the restaurant

0002 The swimming pool and spa are amazing

...

Re: Question about exported data from text parsing in SAS Enterprise Miner

RussAlbright — Wed, 17 Apr 2019 14:41:00 GMT

The Text Parse node creates an underlying representation in the Terms table (which you mentioned you saw) and a term-by-document frequency table that we refer to as the parent table. You cannot directly see this unless you look in your workspace project directory.

When you follow the Text Parse node with a Text Filter node and other Text Mining nodes, these representations are used and not the original input text in that export table. So the stopped terms are being used. It is not until you use a Text Cluster node or a Text Topics node that you see the change on the exported table. And even then, the change is in a set of columns that are the numeric representation of the document (taking into account your stopped terms). The actual raw input text is never changed and exported.

Russ

Re: Question about exported data from text parsing in SAS Enterprise Miner

Jonathanzz — Thu, 18 Apr 2019 04:41:30 GMT

Thanks! Russ.

However, if I want to perform link analysis on review factors which I specified in the start list (i.e. swimming pool, spa, etc.). How should I do it?

Should I edit the dataset to be like this?

review_ID review

0001 I

0001 am

0001 satisfied

0001 with

.... .....

0002 The

0002 swimming

0002 pool

0002 and

0002 spa

.... .....

and let text parsing node to do his job to ignore those word not in the start list.

Re: Question about exported data from text parsing in SAS Enterprise Miner

RussAlbright — Thu, 18 Apr 2019 13:41:24 GMT

Jonathon,

You can use the parent table in the workspace directory. It has the form of triples

termnum document frequency

In the end, in order to interpret results, you just have to map the termnum back to the term string from the terms table.

Russ