BookmarkSubscribeRSS Feed
Nadz
Fluorite | Level 6
Hi, I'd like to split a paragraph into an array of sentences - where delimiter is either . ? ! and consecutive delimiters are treated as one e.g. string=Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week.Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open string1=Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. string2=Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? ... etc Can anyone help?
7 REPLIES 7
RW9
Diamond | Level 26 RW9
Diamond | Level 26

Well, its a bit tricky in code, an example below.  Also you run the risk of a few various, extra spaces, dots in words (for example etc., or e.g.).  I would prefer to proces the text at source rather than at receipt.

data have;
  string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open";
  output;
  string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week.";
  output;
  string="Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia?";
  output;
run;

data want;
  set have;
  length sentance $2000;
  s_ord=1;
  do i=1 to length(string);
    if substr(string,i,1) in ('.','?','!') or i=length(string) then do;
      substr(sentance,s_ord,1)=substr(string,i,1);
      output;
      sentance="";
      s_ord=1;
    end;
    else do;
      substr(sentance,s_ord,1)=substr(string,i,1);
      s_ord=s_ord+1;
    end;
  end;
run;

Nadz
Fluorite | Level 6
Thank you - that's very helpful. Unfortunately, the data is too big to take out of SAS. But how could you create an array? So that instead of having 7 rows in the outpuf file, you still have the 3 rows, but multiple variables in an array string1-stringn? Thanks again
PaigeMiller
Diamond | Level 26

I would think you could use the SCAN function, where delimiters are period, exclamation and question mark, to extract each sentence one by one. Of course, as noted, this fails if there is a period within a sentence, like "Mr. Jones came home."

--
Paige Miller
RW9
Diamond | Level 26 RW9
Diamond | Level 26

Indeed, you raise a good point on scan(), it can reduce the code somewhat:

data want;
  set have;
  i=1;
  do while(scan(string,i,".?!") ne "");
    sentance=scan(string,i,".?!");
    output;
    i=i+1;
  end;
run;

PaigeMiller
Diamond | Level 26

You'd need to add a LENGTH statement before you being the loop

 

LENGTH SENTENCE $ 1024;

 

or some other length.

--
Paige Miller
Nadz
Fluorite | Level 6
Thank you both. A SAS consultant also offered me this which worked - pretty awesome! data have; string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week. Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia? Huge numbers of people heading north from the Mediterranean have created a political crisis in the European Union? Croatian officials said roads leading to the border crossings had also been shut! The crossing on the main road linking Belgrade and Zagreb - at Bajakovo - appeared to be the only one left open"; output; string="Officials said they had no choice after more than 13,000 people entered the country since Hungary fenced off its border with Serbia earlier this week."; output; string="Many have been taken by bus to reception centres but some say they plan to walk to neighbouring Slovenia?"; output; run; data sentences; set have; array arr_sentence {6} $ 200 sentence_1-sentence_6; do n = 1 to 6; arr_sentence{n}=scan(string,n,'.?!'); end; drop n; run;
RW9
Diamond | Level 26 RW9
Diamond | Level 26

Hi,

 

Yes, that is exactly the same as I posted.  Firstly the code is unreadable, need to put things on different lines, use indetation.  There are two main differences, firstly, he has assumed that there will be no more than 6 strings per sentance.  This may or may not work.  Secondly, he uses an array of blocks of text up to a maximum of 6. This may be fine, but I tend to prefer a normalised strcuture (data goes down rather than across) as the code is simplified.

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 7 replies
  • 2907 views
  • 5 likes
  • 3 in conversation