Parse text data into individual words

[crosskonaftw]

Hi SAS community pros!

I am trying to create a data step and/or macro that will parse a large amount of text into individual words for further analysis.

I would like to give SAS a data set with text strings in one variable, and have it return one new variable with all the words from the input variable as separate observations and removing common (or specified list of) delimiters.

I have been experimenting with the SCAN function, but have not had training in how to execute loops or macro array variables.

For example, please see the below excel screenshot for a simplistic example.  Would like to start with SAS data set column A only, and return SAS data set columns A and B

Thank you for your assistance!

[ballardw]

This may get you started. This will duplicate the string. COUNTW and SCAN have a large number of possible combinations of delimiters.

data want;

   string = 'For example, please see the below excel screenshot for a simplistic example.';

   words = countw(string,'sp');

   do i=1 to words;

      word = scan(string,i,'sp');

      output;

   end;

   drop i words;

run;

[Reeza]

Not quite what you asked for, but I'm not sure having the A column as you requested would be useful either.

data have;

a="Wolf, meeting with a Lamb astray from the fold,";output;

a="resolved not to lay violent hands on him"; output;

a="but to find some plea"; output;

run;

data want;

    set have;

    i=1;

    do while (scan(a, i) ne "");

        b=scan(a,i);

        output;

        i+1;

    end;

run;

[CTorres · Accepted solution]

Based on Reeza solution, how about:

data have;

a="Wolf, meeting with a Lamb astray from the fold,"; n+1; output;

a="resolved not to lay violent hands on him"; n+1; output;

a="but to find some plea"; n+1; output;

run;

data temp(keep=b j);

    set have;

    i=1;

    do while (scan(a, i) ne "");

        b=scan(a,i);

       j+1;

        output;

        i+1;

    end;

run;

proc sql noprint;

  create table want (drop=j) as

  select a,b,j

  from have right join temp

  on n=j

  order by j;

quit;

CTorres

[crosskonaftw]

Thank you all for your input!  Apologies for the delayed response as I have been sick and travelling.

I am going to use all of your suggestions as a working starting template and improve upon it.  Thanks again!

[kjohnsonm]

Great question, I needed this exact answer today!