topic Re: how to drop the text variable in texttopic node using text miner in SAS Data Science

how to drop the text variable in texttopic node using text miner

fgfgsgthtgrgrht — Tue, 02 Feb 2021 13:47:08 GMT

I'm using text mining analyses of SAS Enterprise Miner 15.1.

I wonder if we can drop the "text variable" in the "texttopic node", because it always spends a very long period of time for copying the "text variable" from the dataset.

The dataset is around 30GB, the largest size variable belongs to the "text variable", because I'm parsing text data.

If there is a way to drop this variable before creating the texttopic node, that will significantly reduce the required storage space and processing time.

I think it's redundant to copy the "text variable" in the texttopic node, because the terms have been parsed by the "textparsing" and "textfilter" nodes?

my standard procedure of performing text mining analysis:

connect these nodes: sas dataset (with text variable and ID) & textparsing node (with startlist and other settings) & textfilter node (with synonym list and dictionary) & texttopic node (create around 200 single-term topics, no multi-term topic)

Thank you so much!!

Re: how to drop the text variable in texttopic node using text miner

SASJedi — Tue, 02 Feb 2021 17:40:28 GMT

Moved to Analytics -> Data Mining forum for better visibility.

Re: how to drop the text variable in texttopic node using text miner

fgfgsgthtgrgrht — Fri, 05 Feb 2021 10:51:10 GMT

Another related issue is, how to increase the memory and CPU use for the text mining analysis?

Is it adding a SAS code node with the following syntax?

options cpucount=8;
options memsize=1073741824;
options sortsize=1073741824;

Thank you

Re: how to drop the text variable in texttopic node using text miner

gcjfernandez — Sat, 06 Mar 2021 18:19:50 GMT

You could try the DROP variable node from the MODIFY Tab in SAS Enterprise minor to drop this txt variable before connecting the Topics node