topic Re: TF-IDF in SAS Text Miner in SAS Data Science https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175135#M2035 <HTML><HEAD></HEAD><BODY><P>Thanks James for the response. Any idea where I can find the actual Matrix as a SAS dataset .. is it stored under any particular node?</P></BODY></HTML> Thu, 19 Feb 2015 18:38:28 GMT RaviShanbhag 2015-02-19T18:38:28Z TF-IDF in SAS Text Miner https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175133#M2033 <HTML><HEAD></HEAD><BODY><P>I have been looking under the hood at the SAS Text Mining Nodes expecting to find a TF-IDF (Term Frequency - Inverse Document Frequency Matrix) but with no luck. Does anyknow if SAS creates one and stores it somewhere? </P><P></P><P>Any pointers would be great.</P></BODY></HTML> Thu, 19 Feb 2015 17:20:31 GMT https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175133#M2033 RaviShanbhag 2015-02-19T17:20:31Z Re: TF-IDF in SAS Text Miner https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175134#M2034 <HTML><HEAD></HEAD><BODY><P>Absolutely, it does.&nbsp; SAS TM allows you to set the TF parts of the TF-IDF weighting as log, binary or none, and the IDF part as entropy, inverse document frequency, mutual information, or none.&nbsp; The transaction table that comes out of the Text Filter node includes the TF-IDF weightings for each parent term.</P><P></P><P><SPAN style="font-size: 10.0pt; font-family: 'Arial',sans-serif; color: #1f497d;">James A. Cox, Ph.D.</SPAN></P><P><SPAN style="font-size: 10.0pt; font-family: 'Arial',sans-serif; color: #1f497d;">Text Mining Software Development Manager</SPAN></P><P><SPAN style="font-size: 10.0pt; font-family: 'Arial',sans-serif; color: #1f497d;">SAS Institute, Inc.</SPAN></P><P><STRONG style="color: #3366ff; font-size: 13.5pt; font-family: Bookman;"><EM>The Power to Know</EM></STRONG></P><P></P><P><SPAN style="font-size: 11.0pt; font-family: 'Calibri',sans-serif; color: #1f497d;">I wondered why the baseball kept getting bigger. Then it hit me.</SPAN></P></BODY></HTML> Thu, 19 Feb 2015 18:30:16 GMT https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175134#M2034 JamesCoxPhD 2015-02-19T18:30:16Z Re: TF-IDF in SAS Text Miner https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175135#M2035 <HTML><HEAD></HEAD><BODY><P>Thanks James for the response. Any idea where I can find the actual Matrix as a SAS dataset .. is it stored under any particular node?</P></BODY></HTML> Thu, 19 Feb 2015 18:38:28 GMT https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175135#M2035 RaviShanbhag 2015-02-19T18:38:28Z Re: TF-IDF in SAS Text Miner https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175136#M2036 <HTML><HEAD></HEAD><BODY><P>Yep, it is stored in the emws folder for that workspace under the &lt;nodename&gt;_transaction data set.&nbsp; So if this is the first text filter node in the first diagram for the project, it would be emws1.textfilter_transaction .</P><P></P><P>But the easiest way is just to go to the exported data property for the Text Filter node, and explore the transaction table there. </P></BODY></HTML> Thu, 19 Feb 2015 18:53:01 GMT https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175136#M2036 JamesCoxPhD 2015-02-19T18:53:01Z Re: TF-IDF in SAS Text Miner https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175137#M2037 <HTML><HEAD></HEAD><BODY><P>Hello ! Sorry for reviving this post but I'm new in the community and I'm having some problems dealing with this particular issue concerning the TF-IDF matrix.</P><P>The question is: how do I actually use the TF-IDF matrix as the input for the text cluster node? I mean, isn't that what is supposed to happen when you run the text filter node and generate the transaction table ? Does the text cluster node does this by default ?</P><P></P><P>My original post with this issue and pictures can be found here:</P><P><A __default_attr="296926" __jive_macro_name="message" class="jive_macro jive_macro_message" href="https://communities.sas.com/"></A></P><P></P><P>Thanks in advance!</P></BODY></HTML> Mon, 31 Aug 2015 11:34:39 GMT https://communities.sas.com/t5/SAS-Data-Science/TF-IDF-in-SAS-Text-Miner/m-p/175137#M2037 hugo_viga 2015-08-31T11:34:39Z