topic Parse text data into individual words in SAS Programming

Parse text data into individual words

crosskonaftw — Thu, 13 Nov 2014 22:57:16 GMT

Hi SAS community pros!

I am trying to create a data step and/or macro that will parse a large amount of text into individual words for further analysis.

I would like to give SAS a data set with text strings in one variable, and have it return one new variable with all the words from the input variable as separate observations and removing common (or specified list of) delimiters.

I have been experimenting with the SCAN function, but have not had training in how to execute loops or macro array variables.

For example, please see the below excel screenshot for a simplistic example. Would like to start with SAS data set column A only, and return SAS data set columns A and B

Thank you for your assistance!

Re: Parse text data into individual words

ballardw — Thu, 13 Nov 2014 23:09:07 GMT

This may get you started. This will duplicate the string. COUNTW and SCAN have a large number of possible combinations of delimiters.

data want;

string = 'For example, please see the below excel screenshot for a simplistic example.';

words = countw(string,'sp');

do i=1 to words;

word = scan(string,i,'sp');

output;

end;

drop i words;

run;

Re: Parse text data into individual words

Reeza — Thu, 13 Nov 2014 23:10:20 GMT

Not quite what you asked for, but I'm not sure having the A column as you requested would be useful either.

data have;

a="Wolf, meeting with a Lamb astray from the fold,";output;

a="resolved not to lay violent hands on him"; output;

a="but to find some plea"; output;

run;

data want;

set have;

i=1;

do while (scan(a, i) ne "");

b=scan(a,i);

output;

i+1;

end;

run;

Re: Parse text data into individual words

CTorres — Fri, 14 Nov 2014 18:40:47 GMT

Based on Reeza solution, how about:

data have;

a="Wolf, meeting with a Lamb astray from the fold,"; n+1; output;

a="resolved not to lay violent hands on him"; n+1; output;

a="but to find some plea"; n+1; output;

run;

data temp(keep=b j);

set have;

i=1;

do while (scan(a, i) ne "");

b=scan(a,i);

j+1;

output;

i+1;

end;

run;

proc sql noprint;

create table want (drop=j) as

select a,b,j

from have right join temp

on n=j

order by j;

quit;

CTorres

Re: Parse text data into individual words

crosskonaftw — Mon, 24 Nov 2014 08:28:14 GMT

Thank you all for your input! Apologies for the delayed response as I have been sick and travelling.

I am going to use all of your suggestions as a working starting template and improve upon it. Thanks again!

Re: Parse text data into individual words

kjohnsonm — Fri, 18 Mar 2016 16:26:47 GMT

Great question, I needed this exact answer today!