topic Re: Can figure out rudimentary text mining in SAS Programming

Can figure out rudimentary text mining

edouglasa — Thu, 28 Feb 2019 05:36:03 GMT

Hi,

I have never used any of the SAS data mining tools. I have a simple problem. I have 500 legal texts. I want to search to see if they have ten specific words and produce output showing which laws have which texts. Can SAS do this? Can anyone point me to the right platform or code?

Thanks!

Re: Can figure out rudimentary text mining

Satish_Parida — Thu, 28 Feb 2019 05:47:29 GMT

/*Covert the files to long format string*/

data have;
infile cards dlm=',';
input text:$2000.;
cards;
we the people of UK
democracy is by the people
I love egg
;
run;

data want;
set have;
word_flag1=findw(text, 'UK',' ');
word_flag2=findw(text, 'people',' ');
/*.................
word_flag3=findw(text, '****',' ');
word_flag4=findw(text, '****',' ');
..................*/
run;

Re: Can figure out rudimentary text mining

edouglasa — Thu, 28 Feb 2019 05:55:08 GMT

Elegant - thanks!

Re: Can figure out rudimentary text mining

edouglasa — Thu, 28 Feb 2019 05:56:14 GMT

But only one question. Could I read the files from a directly rather than pasting them in?

Re: Can figure out rudimentary text mining

andreas_lds — Thu, 28 Feb 2019 06:37:00 GMT

@edouglasa wrote:

But only one question. Could I read the files from a directly rather than pasting them in?

To be sure that i understood your question: Do you want to read the words you search from file?

Re: Can figure out rudimentary text mining

edouglasa — Thu, 28 Feb 2019 13:40:26 GMT

I'd like to search for terms across word documents or txt files in a directory. I'd like output that indicates everytime which of the search terms is in which word or txt file.

Thanks so much!

Re: Can figure out rudimentary text mining

andreas_lds — Thu, 28 Feb 2019 14:10:00 GMT

@edouglasa wrote:

I'd like to search for terms across word documents or txt files in a directory. I'd like output that indicates everytime which of the search terms is in which word or txt file.

Thanks so much!

I don't think that reading word documents is possible with sas, at least not without wasting an incredible amout of time.

Can your post an example of the expected result-dataset?

Re: Can figure out rudimentary text mining

edouglasa — Thu, 28 Feb 2019 14:18:09 GMT

Got it. Maybe I need to find a more robust datamining software package.

I was looking for an output like:

Text_name, word1, word2, word3

text1, yes, yes, no

text2, yes, no, no

text3, yes, no, no

So the first three texts would all have word1, only text1 would have word 1 and none of the texts would have word3. I was hoping for a way to access the text files from a folder on my machine.

Re: Can figure out rudimentary text mining

andreas_lds — Thu, 28 Feb 2019 14:55:20 GMT

Accessing all text-files in folder is easy:

data work.Matches;
    length 
        filename $ 200 
        text_name $ 1000
        word1-word3 $ 3
    ;

    infile "PATH/*.txt" filename=_filename;
    input;

    text_name = _infile_;
    word1 = ifc(findw(text_name, 'WORD_A'), 'yes', 'no');
    word2 = ifc(findw(text_name, 'WORD_BEE'), 'yes', 'no');
    word3 = ifc(findw(text_name, 'WORD_C'), 'yes', 'no');

    if index(cats(of word:), 'yes') then do;
        filename = _filename;
        output;
    end;
run;

This code is untested. It should create a dataset containing all lines in which at least one word was found.

Re: Can figure out rudimentary text mining

edouglasa — Fri, 08 Mar 2019 17:29:12 GMT

This is amazing. Thanks a ton!