Can figure out rudimentary text mining

[edouglasa]

Hi,

I have never used any of the SAS data mining tools. I have a simple problem. I have 500 legal texts. I want to search to see if they have ten specific words and produce output showing which laws have which texts. Can SAS do this? Can anyone point me to the right platform or code?

Thanks!

[Satish_Parida]

/*Covert the files to long format string*/

data have;
infile cards dlm=',';
input text:$2000.;
cards;
we the people of UK
democracy is by the people
I love egg
;
run;

data want;
set have;
word_flag1=findw(text, 'UK',' ');
word_flag2=findw(text, 'people',' ');
/*.................
word_flag3=findw(text, '****',' ');
word_flag4=findw(text, '****',' ');
..................*/
run;

[edouglasa]

Elegant - thanks!

[edouglasa]

But only one question. Could I read the files from a directly rather than pasting them in?

[andreas_lds]

---

@edouglasa wrote:

But only one question. Could I read the files from a directly rather than pasting them in?

---

To be sure that i understood your question: Do you want to read the words you search from file?

[edouglasa]

I'd like to search for terms across word documents or txt files in a directory. I'd like output that indicates everytime which of the search terms is in which word or txt file.

Thanks so much!

[andreas_lds]

---

@edouglasa wrote:

I'd like to search for terms across word documents or txt files in a directory. I'd like output that indicates everytime which of the search terms is in which word or txt file.

Thanks so much!

---

I don't think that reading word documents is possible with sas, at least not without wasting an incredible amout of time.

Can your post an example of the expected result-dataset?

[edouglasa]

Got it. Maybe I need to find a more robust datamining software package.

I was looking for an output like:

Text_name, word1, word2, word3

text1, yes, yes, no

text2, yes, no, no

text3, yes, no, no

So the first three texts would all have word1, only text1 would have word 1 and none of the texts would have word3. I was hoping for a way to access the text files from a folder on my machine. 

[andreas_lds]

Accessing all text-files in folder is easy:

data work.Matches;
    length 
        filename $ 200 
        text_name $ 1000
        word1-word3 $ 3
    ;

    infile "PATH/*.txt" filename=_filename;
    input;

    text_name = _infile_;
    word1 = ifc(findw(text_name, 'WORD_A'), 'yes', 'no');
    word2 = ifc(findw(text_name, 'WORD_BEE'), 'yes', 'no');
    word3 = ifc(findw(text_name, 'WORD_C'), 'yes', 'no');

    if index(cats(of word:), 'yes') then do;
        filename = _filename;
        output;
    end;
run;

This code is untested.  It should create a dataset containing all lines in which at least one word was found.

[edouglasa · Accepted solution]

This is amazing. Thanks a ton!