topic Re: How to detect and treat apostrophe and foreign characters in a string? in SAS Programming

How to detect and treat apostrophe and foreign characters in a string?

xyxu — Mon, 24 Oct 2022 04:32:32 GMT

My string variable contains apostrophes and foreign characters in a few observations. For example,

data have;
	input var $30.;
	datalines;
Let's go for dinner
pi�ata dance collective
Another 인천
The 4th line
;
run;
I want to remove the apostrophe (') in the first observation, and delete the second and third observations because they contain characters that are neither alphabet nor number. Is there a good way to do this in SAS?

Re: How to detect and treat apostrophe and foreign characters in a string?

LinusH — Mon, 24 Oct 2022 07:04:10 GMT

You can use the COMPRESS function to remove certain characters.

Then there's a lot of function to identify different type of characters in a string:

https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lefunctionsref/n01f5qrjoh9h4hn1olbdpb5pr2td.htm#n0epkw2tu7yuion1877mn8mjjrex

I would try a combination of the functions starting with NOT...

Re: How to detect and treat apostrophe and foreign characters in a string?

Patrick — Mon, 24 Oct 2022 07:12:03 GMT

The weird characters you're getting are likely due to using a single byte encoded editor or environment for multibyte characters (i.e. UTF-8).

Below should satisfy what you're asking for.

data want;
  set have;
  var=compress(var,"'");
  if findc(var,' ','kfnp') then delete;
run;

Re: How to detect and treat apostrophe and foreign characters in a string?

Ksharp — Mon, 24 Oct 2022 11:51:42 GMT

data want;
 set have;
 var=compress(var,"'");
 if prxmatch('/[^a-z\d\s]/i',var) then delete;
run;

Re: How to detect and treat apostrophe and foreign characters in a string?

xyxu — Mon, 24 Oct 2022 22:08:44 GMT

Thanks! Do you have any suggestions for improving the editor's encoding issues? I used SAS 9.4 (English). If I add encoding='utf-8' into the import code, I got

ERROR: Invalid string.
FATAL: Unrecoverable I/O error detected in the execution of the DATA step program.
Aborted during the EXECUTION phase.