topic Re: Getting false positives when flagging for words in SAS Data Science

Getting false positives when flagging for words

brulard — Fri, 10 Mar 2017 21:00:13 GMT

I'm trying to scan a column (using base SAS) that contains notations that agents enter when dealing with customers. The problem is that i am getting many false positives. That is, when coding a string of words to flag, I get records that flag 1, yet without the words being in the value.

I tried using both index function and the prxmatch function, with same result. The column that I am scanning is formatted as $2000.

Example of my query:

If you have a suggestion as an alternate way to flag for presence of words in given field, with better accuracy, please advise.

thank you

Re: Getting false positives when flagging for words

ballardw — Fri, 10 Mar 2017 21:19:33 GMT

You should also show at least one example of the data that is incorrectly flagged.

Re: Getting false positives when flagging for words

brulard — Fri, 10 Mar 2017 21:21:26 GMT

Hi Ballardw, here is an example:

MDECLINED LOC INCR TO 4K$ FOR X-MAS SHOPPING, SCORE 684/3/471, EST ON BURO SINCE 1990, NO DEROGS ON BURO, DEBT RATIO HIGH, HIGHEST TRADE 18K$ ALL MAX ON OTHER TRADE, GOOD PYMT HISTORY.

Re: Getting false positives when flagging for words

FriedEgg — Fri, 10 Mar 2017 21:26:52 GMT

Assuming you issue is finding substrings instead of whole words try adding \b to signify a word boundry on both sides of your term list

/\b(COUNTER|COUNTERS|CW30|REISSUE|REISSUED|STRATEGY|RE-OPEN|RE-ISSUE|APPROVE|PIN|SECURITY|TRANSACTION|TRANSACTIONS)\b/i

Re: Getting false positives when flagging for words

Shmuel — Fri, 10 Mar 2017 21:58:29 GMT

You can use alternative way.:

data _NULL_;
     column_have = "MDECLINED LOC INCR TO 4K$ FOR X-MAS SHOPPING, SCORE 684/3/471, EST ON BURO SINCE 1990, NO DEROGS ON BURO, DEBT RATIO HIGH, HIGHEST TRADE 18K$ ALL MAX ON OTHER TRADE, GOOD PYMT HISTORY.";
     chk_for ="COUNTER|COUNTERS|CW30|REISSUE|REISSUED|STRATEGY|RE-OPEN|RE-ISSUE|APPROVE|PIN|SECURITY|TRANSACTION|TRANSACTIONS";
     do i=1 to 50;
        word = scan(chk_for,i,'|');
        if word = ' ' then leave;
        pos = findw(column_have,word); put word= pos=; 
     end;
RUN;

none of the words in chk_for variable were found. All have position 0.

Re: Getting false positives when flagging for words

brulard — Fri, 10 Mar 2017 23:44:23 GMT

thanks for the tip... i ll try it on Monday

Re: Getting false positives when flagging for words

brulard — Mon, 13 Mar 2017 11:32:33 GMT

thanks... issue is primarily finding whole word match... not sure why I am getting false positves

Re: Getting false positives when flagging for words

brulard — Mon, 13 Mar 2017 16:40:26 GMT

ok, I think i figured it out. The result I was getting, not true false postives... but is the result of my string |PIN| which flagged word SHOPPING. To avoid hitting this, i could add a space before and after, | PIN |. So i think this closes this message