https://communities.sas.com/t5/SAS-Data-Science/term-document-frequency-matrix-from-textmine-is-ignoring-small/m-p/943140#M10920 <P>On the PARSE statement, please specify :</P> <UL class="lia-list-style-type-square"> <LI><STRONG>NONOUNGROUPS</STRONG> | NONG&nbsp; :&nbsp; Suppresses noun group extraction in parsing</LI> <LI><STRONG>NOSTEMMING</STRONG>&nbsp; :&nbsp; Suppresses stemming in parsing</LI> <LI><STRONG>NOTAGGING</STRONG>&nbsp; :&nbsp; Suppresses part-of-speech tagging in parsing</LI> <LI><STRONG>SHOWDROPPEDTERMS</STRONG>&nbsp; :&nbsp; Includes dropped terms in the OUTTERMS= data table</LI> </UL> <P>Still the same problem?&nbsp;<BR />Are you sure the 1 or 2 words in the small documents (the ones you are missing in the output tables) are not in the stop list?</P> <P><BR />BR, Koen</P> Mon, 09 Sep 2024 13:43:55 GMT sbxkoenk 2024-09-09T13:43:55Z https://communities.sas.com/t5/SAS-Data-Science/term-document-frequency-matrix-from-textmine-is-ignoring-small/m-p/942419#M10918 <P>I have used textmine to find the frequencies of various words for each of a set of documents. Each document is a free text field that consists of one or more words- this free text field corresponds to one of the columns. There is also an Index field that indicates the document number along with various other columns.</P><P>&nbsp;</P><P>I used the following code to output the term-document matrix:</P><P>&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="William29_0-1725423083486.png" style="width: 400px;"><img src="https://communities.sas.com/t5/image/serverpage/image-id/99945i087A99B9CC09CC28/image-size/medium?v=v2&amp;px=400" role="button" title="William29_0-1725423083486.png" alt="William29_0-1725423083486.png" /></span></P><P>I noticed that when looking at the term document matrix (the file mentioned in outchild), that if I search for the 'Index' column value for a document with or one (and sometimes two words) within the 'Document' column of the term-document frequency matrix, that often it cannot be found at all. It appears that textmine is not even processing this columns.</P><P>&nbsp;</P><P>Is this a known feature of textmine (is it supposed to be doing this)?</P><P>&nbsp;</P><P>Is there a simple option within textmine to stop it doing this (as opposed to editing the documents and adding a lot of place-filler words to increase the document length)?</P><P>&nbsp;</P><P>I am wondering whether this might have to do with the fact that the algorithm cannot identify whether the words are verbs or nouns (which is something the algorithm does) when there are very few words and so it ignores the document altogether?</P> Wed, 04 Sep 2024 04:20:28 GMT https://communities.sas.com/t5/SAS-Data-Science/term-document-frequency-matrix-from-textmine-is-ignoring-small/m-p/942419#M10918 William29 2024-09-04T04:20:28Z https://communities.sas.com/t5/SAS-Data-Science/term-document-frequency-matrix-from-textmine-is-ignoring-small/m-p/943140#M10920 <P>On the PARSE statement, please specify :</P> <UL class="lia-list-style-type-square"> <LI><STRONG>NONOUNGROUPS</STRONG> | NONG&nbsp; :&nbsp; Suppresses noun group extraction in parsing</LI> <LI><STRONG>NOSTEMMING</STRONG>&nbsp; :&nbsp; Suppresses stemming in parsing</LI> <LI><STRONG>NOTAGGING</STRONG>&nbsp; :&nbsp; Suppresses part-of-speech tagging in parsing</LI> <LI><STRONG>SHOWDROPPEDTERMS</STRONG>&nbsp; :&nbsp; Includes dropped terms in the OUTTERMS= data table</LI> </UL> <P>Still the same problem?&nbsp;<BR />Are you sure the 1 or 2 words in the small documents (the ones you are missing in the output tables) are not in the stop list?</P> <P><BR />BR, Koen</P> Mon, 09 Sep 2024 13:43:55 GMT https://communities.sas.com/t5/SAS-Data-Science/term-document-frequency-matrix-from-textmine-is-ignoring-small/m-p/943140#M10920 sbxkoenk 2024-09-09T13:43:55Z