topic Re: Splitting a paragraph into an array of sentences in SAS Procedures

Splitting a paragraph into an array of sentences

Nadz — Fri, 18 Sep 2015 09:46:25 GMT

Hi, I'd like to split a paragraph into an array of sentences - where delimiter is either . ? ! and consecutive delimiters are treated as one e.g. string=Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week.Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open string1=Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. string2=Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? ... etc Can anyone help?

Re: Splitting a paragraph into an array of sentences

RW9 — Fri, 18 Sep 2015 10:20:51 GMT

Well, its a bit tricky in code, an example below. Also you run the risk of a few various, extra spaces, dots in words (for example etc., or e.g.). I would prefer to proces the text at source rather than at receipt.

data have;
string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open";
output;
string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week.";
output;
string="Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia?";
output;
run;

data want;
set have;
length sentance $2000;
s_ord=1;
do i=1 to length(string);
if substr(string,i,1) in ('.','?','!') or i=length(string) then do;
substr(sentance,s_ord,1)=substr(string,i,1);
output;
sentance="";
s_ord=1;
end;
else do;
substr(sentance,s_ord,1)=substr(string,i,1);
s_ord=s_ord+1;
end;
end;
run;

Re: Splitting a paragraph into an array of sentences

Nadz — Fri, 18 Sep 2015 11:58:06 GMT

Thank you - that's very helpful. Unfortunately, the data is too big to take out of SAS. But how could you create an array? So that instead of having 7 rows in the outpuf file, you still have the 3 rows, but multiple variables in an array string1-stringn? Thanks again

Re: Splitting a paragraph into an array of sentences

PaigeMiller — Fri, 18 Sep 2015 12:31:39 GMT

I would think you could use the SCAN function, where delimiters are period, exclamation and question mark, to extract each sentence one by one. Of course, as noted, this fails if there is a period within a sentence, like "Mr. Jones came home."

Re: Splitting a paragraph into an array of sentences

RW9 — Fri, 18 Sep 2015 13:00:53 GMT

Indeed, you raise a good point on scan(), it can reduce the code somewhat:

data want;
set have;
i=1;
do while(scan(string,i,".?!") ne "");
sentance=scan(string,i,".?!");
output;
i=i+1;
end;
run;

Re: Splitting a paragraph into an array of sentences

PaigeMiller — Fri, 18 Sep 2015 13:40:35 GMT

You'd need to add a LENGTH statement before you being the loop

LENGTH SENTENCE $ 1024;

or some other length.

Re: Splitting a paragraph into an array of sentences

Nadz — Mon, 21 Sep 2015 08:51:39 GMT

Thank you both. A SAS consultant also offered me this which worked - pretty awesome! data have; string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open"; output; string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week."; output; string="Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia?"; output; run; data sentences; set have; array arr_sentence {6} $ 200 sentence_1-sentence_6; do n = 1 to 6; arr_sentence{n}=scan(string,n,'.?!'); end; drop n; run;

Re: Splitting a paragraph into an array of sentences

RW9 — Mon, 21 Sep 2015 08:56:26 GMT

Hi,

Yes, that is exactly the same as I posted. Firstly the code is unreadable, need to put things on different lines, use indetation. There are two main differences, firstly, he has assumed that there will be no more than 6 strings per sentance. This may or may not work. Secondly, he uses an array of blocks of text up to a maximum of 6. This may be fine, but I tend to prefer a normalised strcuture (data goes down rather than across) as the code is simplified.