Text Mining Using Base SAS

[Mgarret]

Hi All,

I'm trying to right a program which will sum up all the unique words in a variable -  this needs to be not case sensitive.  I need to use just Base SAS.

CONTENT

signs that I might be experiencing Candida? A. Here is our list of 100 Common Candida Symptoms associated with systemic. Use zyrtec

i    a week of sttn.. about 3 weeks ago.. LO had bad allergies so he had to take some zyrtec but he slept sooo good

Allergy Aid Cleansing Expert services Are Necessary Some folks are incapable of having a great night's sleep

So the output I'm looking for is something like this (this is not the full list):

Word	Count
zyrtec	2
Some	2
of	3
I	2
he	2
had	2
experiencing	1
Common	1
Cleansing	1
Candida	1
a	2

Is there a way to write a do loop? It would obviously way too difficult  to you just string functions to do this.

Any assistance is greatly appreciated.

Thanks! 

[Reeza]

A few functions - countw, compress, scan, and lowcase can get you pretty far.

data words;

set have;

num_words=countw(sentence);

do i=1 to num_words;

word=lowcase(compress(scan(sentence, i), , 'ka'));

output;

end;

keep word;

run;

proc freq data=words;

table word;

run;