topic Re: Need a way to identify non-USA keyboard characters in a string in SAS Programming

Need a way to identify non-USA keyboard characters in a string

biglerc — Tue, 21 Jan 2025 18:26:58 GMT

Hello Community!

I have data that gets aggregated from multiple countries and I need a way to create a flag for a character variable that contains characters that are not found on USA keyboards. Is there a way to do this? Any guidance is appreciated!

Thanks!

Re: Need a way to identify non-USA keyboard characters in a string

PaigeMiller — Tue, 21 Jan 2025 18:35:05 GMT

Lots of answers here, depending on what "non-USA keyboard characters" really means to you.

https://communities.sas.com/t5/SAS-Programming/Detect-or-Exclude-rows-which-include-not-standard-ASCII/td-p/886866

Re: Need a way to identify non-USA keyboard characters in a string

biglerc — Mon, 27 Jan 2025 16:40:03 GMT

Thank you very much for your response! I ended up using a loop through ascii numbers and it worked.

Re: Need a way to identify non-USA keyboard characters in a string

Patrick — Mon, 27 Jan 2025 22:36:44 GMT

@biglerc If you're only dealing with single byte characters then below will also work and requires less code.

data have;
	var='AbIöIxz'; output;
	var='AbIoIxz'; output;
run;

data want;
	set have;
	nonEnglishChar_flg= prxmatch('/[^a-z]/i',strip(var))>0;
run;

proc print data=want;
run;

Re: Need a way to identify non-USA keyboard characters in a string

biglerc — Mon, 27 Jan 2025 22:45:43 GMT

Thank you, Patrick! I am dealing with loads of them, since the data is coming from different countries. Here's an example: RESULT = ×ª×©×•×‘×” ×œ×“×•×’×ž×”.

My code currently uses substr to go character by character in every text string and identifies the position of a special character (if it finds one). This is working perfectly except for the instances like my example above where the entire string is special characters. In these cases SAS throws an error. Here is my code - it may be too out of context but maybe you can follow it? I'm just wondering why it won't do a substr if the whole value is special characters.

/* step through each character in the field to look for special character(s) */
do i = 1 to length(current_variable) ;

if rank(substr(current_variable),i,1)) gt 0 and rank(substr(current_variable),i,1)) lt 32 or rank(substr(current_variable),i,1)) gt 127 then do;
position= i;
char = byte(rank(substr(varvalue,i,1)));
nonascii_pos = catx(',',nonascii_pos,strip(put(position,8.)));
nonascii = catx(',',nonascii,strip(char ));
end;
end;

Thanks for taking the time to answer my question!

Re: Need a way to identify non-USA keyboard characters in a string

Tom — Mon, 27 Jan 2025 22:48:28 GMT

@biglerc wrote:

Thank you very much for your response! I ended up using a loop through ascii numbers and it worked.

No need to loop. You can use COLLATE to generate them. All of the printable characters, from space to tilde, can be generated with COLLATE(32,126). Then use INDEXC() or VERIFY() function depending whether you want to test for the existence of of valid characters or existence of invalid characters.

data want;
   set have;
   any_valid = 0 ne indexc(string,collate(32,126));
   any_invalid = 0 ne verify(string,collate(32,126));
run;

Example:

data have;
  input string $50.;
  if string='alpha' then string=tranwrd(string,'alpha','CEB1'x);
cards4;
`1234567890-=~!@#$%^&*()_+
qwertyuiop[]\QWERTYUIOP{}|
asdfghjkl;'ASDFGHJKL:"
zxcvbnm,./ZXCVBNM<>?
alpha
;;;;

data want;
   set have;
   any_valid = 0 ne indexc(trim(string),collate(32,126));
   any_invalid = 0 ne verify(trim(string),collate(32,126));
run;

Result

Re: Need a way to identify non-USA keyboard characters in a string

biglerc — Mon, 27 Jan 2025 22:50:52 GMT

This is fantastic! I have never used COLLATE so this is a great learning experience for me. Thank you sooo much!

Re: Need a way to identify non-USA keyboard characters in a string

biglerc — Mon, 27 Jan 2025 23:12:02 GMT

Hi Tom,

Might I ask you for guidance on a slight modification to the code you sent? Part of my requirement is to keep track of the position of the special character in a string. So for example, if the variable called "RESULT" contains 5 characters and the 3rd character is a special character, then I need to output 1) the name of the variable, 2) the special character(s) and 3) the position(s) of the special character(s). This is why I was looping through each character in the string. My code works correctly if there is at least one NON-special character in the string, however, if all characters are special characters, it crashes.

I'm currently experimenting to see if I can combine my looping with the collate you taught me to resolve this 🙂

Thanks!

Re: Need a way to identify non-USA keyboard characters in a string

Tom — Tue, 28 Jan 2025 13:56:28 GMT

Looping over the characters in a list should work.

You might share your issue with the loop failing when there are no special characters in a new topic. Provide a couple of simple example strings that are causing your problems.

Watch out if you are running with system encoding set to UTF-8 or other multibyte character encodings. It wouldn't matter for generating the simple binary flag variable since the normal 7bit ASCII characters only use one byte. But for locating other characters you will need to use the Kxxxx series of functions that are designed for working with characters that might use more than one byte.

So perhaps something like:

if verify(string,collate(32,126) then do;
  do i=1 to klength(string);
    length char $4 ;
    char = ksubstr(string,i,1);
    if verify(char,collate(32,126)) then put i= char= ;
  end;
end;

Re: Need a way to identify non-USA keyboard characters in a string

baharvey7817_1 — Thu, 06 Nov 2025 21:01:03 GMT

Will this code work in a mainframe (z/OS) environment? I've tried using the NOTALNUM function but it's interpreting accented characters as alphanumeric anyway. The locale is en_US and the encoding is OPEN_ED-1047.

Re: Need a way to identify non-USA keyboard characters in a string

Tom — Thu, 06 Nov 2025 21:45:49 GMT

OPEN_ED-1047 is a Single Byte encoding. So you don't need to worry about multi-byte characters or using KSUBSTR().

You can use KCVT() to convert the ASCII string generated by COLLATE into the equivalent OPEN_ED-10247 string.

  do i=1 to length(string);
    char = char(string,i);
    if verify(char,kcvt(collate(32,126),'latin1','OPEN_ED-1047')) then do;
      put i= char= $hex2. ;
    end;
  end;

Re: Need a way to identify non-USA keyboard characters in a string

baharvey7817_1 — Fri, 07 Nov 2025 04:13:04 GMT

Thanks for responding!

The issue I've been running into on z/OS is that despite using the NOTALNUM() function or using PRXMATCH( '/[^a-zA-Z0-9]/','string') to exclude accented characters it treats them as alphanumeic. Really frustraing! I want to flag or show the position of these characters in the variable in the data.
Any ideas?

Re: Need a way to identify non-USA keyboard characters in a string

Ksharp — Fri, 07 Nov 2025 06:21:03 GMT

You could also try to use COMPRESS().

x=compress(string,,'kad');

If x is not missing ,there are some characters you want to look for.