topic Re: Need to use COMPGED or COMPLEV function on multi-byte text data b ـب ـبـ in SAS Programming

Need to use COMPGED or COMPLEV function on multi-byte text data

TomKari — Sat, 04 Aug 2018 23:45:41 GMT

Hi, all

I need to use something like the "compged" or "complev" function to compare two text strings, but I need to process UTF-8 data containing weird and wild characters. The SAS NLS guides say that these two functions aren't certified for multi-byte character data.

Does anybody have any suggestions for how I can do this?

Much thanks,
Tom

Re: Need to use COMPGED or COMPLEV function on multi-byte text data b ـب ـبـ

ChrisNZ — Sun, 05 Aug 2018 02:07:16 GMT

I guess you'd have to compute the distance yourself using the k* functions.

1. That's good ballot entry

2. Choices must be made as that's not a straight forward computation. What is the distance between 'hä' and 'hà' ?

3. Syllabic scripts or ideograms would provide nice head-scratchers (though there may already be algorithms for these).

Even comparisons of alphabetic scripts like Arabic would not be easy as the character changes depending on the position.

ـب ـبـ بـ ب are all letter B.

Re: Need to use COMPGED or COMPLEV function on multi-byte text data

art297 — Sun, 05 Aug 2018 03:25:26 GMT

@TomKari: Do a google search for: generalized edit distance utf-8 r

There are a number of r packages available.

Art, CEO, AnalystFinder.com

Re: Need to use COMPGED or COMPLEV function on multi-byte text data

PGStats — Sun, 05 Aug 2018 04:54:08 GMT

Look as the BASECHAR function in NLS. Without the second argument, it returns an ASCII version of your string. At least, that's what the documentation example suggests.