topic Re: Text Search in SAS Programming

Text Search

SASPhile — Thu, 09 Feb 2017 17:07:16 GMT

There is a credit hist coulmn that gives information of customer. I want to look at records whose credit is based based on some key terms like

poor,bad,low, delinquent and in various forms. a like operator doesnt serve the purpose as sometimes the words can be shortned by the representative. can this be done in regex, so as i look through the cases i can add the new words in search?

Re: Text Search

Cynthia_sas — Thu, 09 Feb 2017 17:50:47 GMT

Hi:
This sounds like the kind of thing that Enterprise Miner and Text Miner can do. What have you tried? What tools do you have?

cynthia

Re: Text Search

SASPhile — Thu, 09 Feb 2017 17:55:07 GMT

Base SAS

Re: Text Search

FriedEgg — Thu, 09 Feb 2017 18:31:02 GMT

data terms;
input term : $8. @@;
cards;
poor bad low negative neg lw
;
run;
%let terms=&sysnobs;
%put &=terms;

proc transpose data=terms out=terms_array(drop=_name_) prefix=term;
var term;
run;

data have;
str='Lady shows poor judgement.  Has good credit.';
output;
str='Good Credit';
output;
str='This guy has terrible credit!';
output;
str='Poor credit';
output;
run;

data want;
set terms_array;
array term[&terms];
drop term:;

do until (done);
  set have end=done;
  do _n_=1 to countw(str);
    flag=(whichc(lowcase(scan(str,_n_)),of term[*])>0);
	if flag>0 then leave;
  end;
  output;
end;
stop;
run;

Re: Text Search

anoopmohandas7 — Thu, 09 Feb 2017 19:19:54 GMT

I might be over oversimplifying. But if you could look at the data and see if they all start with a disticnt letter that we could be associated with a status we can format it and achieve what you look for. But as I speak this I have no idea of your data.

data test ;
input id status $ ;
datalines ;
12 Poor
13 bad
15 Del
16 bd
19 lw
;
run ;

proc sql ;
select id, 
case 
  when upcase(substr(status,1,1)) = 'P' then 'POOR'
  when upcase(substr(status,1,1)) = 'B' then 'BAD'
  when upcase(substr(status,1,1)) = 'G' then 'GOOD'
  when upcase(substr(status,1,1)) = 'D' then 'DELINQUENT'
  when upcase(substr(status,1,1)) = 'L' then 'LOW'
  else 'OTHER'
end as status1 
from test ;
quit ;

Re: Text Search

SASPhile — Tue, 14 Feb 2017 15:19:51 GMT

it is not a simple search with one word values in it. It is a text column with 5000 plus characters and need to identify if any of the expected words are present in the text.

Re: Text Search

SASPhile — Wed, 15 Feb 2017 19:02:37 GMT

if there are phrases or compund words like "End of period" or "Introductory APR"?

these words will be passed as indvidual words and not as part of the string that I'm insterested in.

Re: Text Search

Patrick — Wed, 15 Feb 2017 22:25:33 GMT

May be something like below could do the job for you.

data search_terms;
  infile datalines truncover;
  input search_term $100.;
  search_term=lowcase(search_term);
  datalines;
End of period
Introductory APR
Good
Poor
;
run;

data have;
  infile datalines truncover;
  input sentence $200.;
  datalines;
Lady shows poor judgement.  Has good credit.
Good Credit
This guy has terrible credit!
Poor credit
blah end of period blah
;
run;

data want;
  set have end=last;
  do _i=1 to nobs;
    set search_terms nobs=nobs point=_i;
    if find(sentence,strip(search_term),'it') then leave;
    call missing(search_term);
  end;
run;

Re: Text Search

Peter_C — Wed, 15 Feb 2017 23:27:04 GMT

Perhaps sum the weights created when a specific phrase is found in the long text.
It could separate searches
- looking for indications of "good"
- looking for indications of "poor"
Might make a significant cpu impact