Help using Base SAS procedures

Splitting a paragraph into an array of sentences

Reply
New Contributor
Posts: 3

Splitting a paragraph into an array of sentences

Hi, I'd like to split a paragraph into an array of sentences - where delimiter is either . ? ! and consecutive delimiters are treated as one e.g. string=Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week.Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open string1=Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. string2=Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? ... etc Can anyone help?
Super User
Super User
Posts: 7,997

Re: Splitting a paragraph into an array of sentences

Well, its a bit tricky in code, an example below.  Also you run the risk of a few various, extra spaces, dots in words (for example etc., or e.g.).  I would prefer to proces the text at source rather than at receipt.

data have;
  string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open";
  output;
  string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week.";
  output;
  string="Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia?";
  output;
run;

data want;
  set have;
  length sentance $2000;
  s_ord=1;
  do i=1 to length(string);
    if substr(string,i,1) in ('.','?','!') or i=length(string) then do;
      substr(sentance,s_ord,1)=substr(string,i,1);
      output;
      sentance="";
      s_ord=1;
    end;
    else do;
      substr(sentance,s_ord,1)=substr(string,i,1);
      s_ord=s_ord+1;
    end;
  end;
run;

New Contributor
Posts: 3

Re: Splitting a paragraph into an array of sentences

Thank you - that's very helpful. Unfortunately, the data is too big to take out of SAS. But how could you create an array? So that instead of having 7 rows in the outpuf file, you still have the 3 rows, but multiple variables in an array string1-stringn? Thanks again
Trusted Advisor
Posts: 1,934

Re: Splitting a paragraph into an array of sentences

I would think you could use the SCAN function, where delimiters are period, exclamation and question mark, to extract each sentence one by one. Of course, as noted, this fails if there is a period within a sentence, like "Mr. Jones came home."

Super User
Super User
Posts: 7,997

Re: Splitting a paragraph into an array of sentences

Posted in reply to PaigeMiller

Indeed, you raise a good point on scan(), it can reduce the code somewhat:

data want;
  set have;
  i=1;
  do while(scan(string,i,".?!") ne "");
    sentance=scan(string,i,".?!");
    output;
    i=i+1;
  end;
run;

Trusted Advisor
Posts: 1,934

Re: Splitting a paragraph into an array of sentences

[ Edited ]

You'd need to add a LENGTH statement before you being the loop

 

LENGTH SENTENCE $ 1024;

 

or some other length.

New Contributor
Posts: 3

Re: Splitting a paragraph into an array of sentences

[ Edited ]
Posted in reply to PaigeMiller
Thank you both. A SAS consultant also offered me this which worked - pretty awesome! data have; string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open"; output; string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week."; output; string="Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia?"; output; run; data sentences; set have; array arr_sentence {6} $ 200 sentence_1-sentence_6; do n = 1 to 6; arr_sentence{n}=scan(string,n,'.?!'); end; drop n; run;
Super User
Super User
Posts: 7,997

Re: Splitting a paragraph into an array of sentences

Hi,

 

Yes, that is exactly the same as I posted.  Firstly the code is unreadable, need to put things on different lines, use indetation.  There are two main differences, firstly, he has assumed that there will be no more than 6 strings per sentance.  This may or may not work.  Secondly, he uses an array of blocks of text up to a maximum of 6. This may be fine, but I tend to prefer a normalised strcuture (data goes down rather than across) as the code is simplified.

Ask a Question
Discussion stats
  • 7 replies
  • 490 views
  • 5 likes
  • 3 in conversation