topic Re: How to eliminate articles,prepositions and pronous from data in SAS Data Science

How to eliminate articles,prepositions and pronous from data

tlnarayana26 — Thu, 18 Jun 2015 12:42:11 GMT

Every one

i have data set

word count

lakshmi 10

ads 6

market 5

the 4

to 2

laks 2

is 2

what 2

and 1

help 1

book 1

How to eliminate articles,prepositions and pronous from above data set with out the word.

using base sas

plz help me.

Re: How to eliminate articles,prepositions and pronous from data

RobertWoodruff — Fri, 19 Jun 2015 19:26:36 GMT

You want to scan this dataset and remove records where the
variable “word” is an articles, prepositions or pronoun?

Re: How to eliminate articles,prepositions and pronous from data

ballardw — Fri, 19 Jun 2015 19:54:07 GMT

If the list isn't real long this is a simple way for your example data:

data have;
input word $    count;
datalines;
lakshmi     10
ads            6
market       5
the            4
to              2
laks           2
is              2
what         2
and           1
help          1
book         1
;
run;

data want;
set have;
if upcase(word) in ('THE','A','AN','I','HE','SHE','WE','IT','THEM','TO','AND') then delete;
run;

I use upcase because data could have "The" and "the". Add words as desired. If the list gets real long then creating a data set of unwanted words and a Proc SQL approach might be better.

Re: How to eliminate articles,prepositions and pronous from data

Reeza — Fri, 19 Jun 2015 20:47:59 GMT

Look up Natural Language Processing - NLP and see if you can find a list of words that would be considered articles, prepositions, nouns. Then its a simple SQL query to remove them.

Here's a list of 'stop words'

http://jmlr.org/papers/volume5/lewis04a/a11-smart-stop-list/english.stop

Re: How to eliminate articles,prepositions and pronous from data

tlnarayana26 — Mon, 22 Jun 2015 06:03:02 GMT

Thanks for all your Great help.

Hi reeza sir above link gives list words. how can move to next step. plz help.

Re: How to eliminate articles,prepositions and pronous from data

Reeza — Mon, 22 Jun 2015 07:21:12 GMT

Well, for starters it's not sir.

Read in the list from the link <- RemoveList.

Create your word list <- WordList

Remove words from WordList via SQL step:

proc sql;

create table WordList2 as

select *

from WordList

where word NOT IN (select word from RemoveList)

order by word;

quit;