topic Re: Stop Words in SAS Programming

Stop Words

mar00390 — Wed, 15 Jan 2020 19:21:52 GMT

How do I delete prepositions/conjunctions/auxiliary verbs from a string? My strings have a length of 32,767

Re: Stop Words

SASKiwi — Wed, 15 Jan 2020 19:49:53 GMT

Base SAS doesn't contain any functionality to identify language components. You are limited to word and character pattern matches.

SAS Text Miner probably has more capabilities, but I doubt it can parse grammatical terms.

Re: Stop Words

Reeza — Wed, 15 Jan 2020 19:55:49 GMT

You need to use the text parsing node with a stop list.
https://documentation.sas.com/?docsetId=tmref&docsetTarget=p0h83e35apv01tn13y0ulqr9r2tp.htm&docsetVersion=14.3&locale=en

Re: Stop Words

ErikLund_Jensen — Wed, 15 Jan 2020 19:55:54 GMT

Hi @mar00390

You nede to create a list of stop words. There might be something to download as a starting point, but otherwise it's just hard work. Given the list and a SAS data set with your strings, an easy solution is to use a format to pick the stop words in the string. The following working code shows the principles.

You need to set proper lengths etc. to make it work with your data. Given your word classes it is probably unnecessary to handle uppercase/lowercase words, but it can be done with a lowcase function on teststr. And be aware that words in the output string are always separated by one blank even if there are more in the input string.

* Test data;
data stopwords;
	input stopword $20.;
	cards;
abc 
xyz 
;
run;

data have;
	infile cards truncover;
	input string $char200.;
	cards;
aaa abc bbbbbbbbbb c 123 dddd ffff xyz
123 zzzzzzzzzz xyz hhhhhh
;
run;

* Create format;
data stopfmt; set stopwords end=end;
	retain type 'C' fmtname 'stopfmt';
	start = stopword;
	label = stopword;
	output;
	if end then do;
		hlo = 'O';
		start = '';
		label = '';
		output;
	end;

run;
proc format cntlin=stopfmt;
run;

* Remove all words defined as stop words from string;
data want (drop=i teststr); set have;
	length newstr $200 teststr $50;
	do i = 1 to countw(string,' ');
		teststr = scan(string,i,' ');
		if put(teststr,$stopfmt.) = '' then newstr = catx(' ',newstr,teststr);
	end;
run;

Re: Stop Words

ErikLund_Jensen — Wed, 15 Jan 2020 19:58:48 GMT

A smart guy with a better command of hash objects would give a more elegant solution without the format step.

Re: Stop Words

Reeza — Wed, 15 Jan 2020 20:02:06 GMT

Pretty sure the OP is using TextMiner though.

Re: Stop Words

mar00390 — Thu, 16 Jan 2020 14:17:04 GMT

This kept it the same. Is there a reason it wouldn't work?

Re: Stop Words

ErikLund_Jensen — Thu, 16 Jan 2020 15:18:16 GMT

Hi @mar00390

Sorry, but I need to know a bit more to answer that.

If you ran my example code, then notice that the original string is also kept in output, so you have before/after in each record.

If you used your own data, I need to have an example, at least one stop word and a string where the stop word occurs. Then I'll look into it.