topic Re: Detect or Exclude rows which include not standard ASCII characters in SAS Programming

Detect or Exclude rows which include not standard ASCII characters

dcortell — Fri, 28 Jul 2023 17:28:24 GMT

Hi Experts

I have the following sample dataset

data have;
length url $3000.;
input url;
datalines;
blogs.sas.com/wan/2022/03/18/sas-eg-è·³å‡ºéŒ¯èª¤è¨Š 
www.dog.it
;
run;

I'm trying to find a way to exclude all the row in the dataset which include not ASCII standard characters or not printable characters. Any hints appreciated

Re: Detect or Exclude rows which include not standard ASCII characters

FreelanceReinh — Fri, 28 Jul 2023 18:15:52 GMT

Hi @dcortell,

You can use the FINDC function with modifiers corresponding to character classes that you may want to keep or exclude. Or the VERIFY function:

data want;
set have;
if ~verify(url, collate(32,126));
run;

In this example I use the COLLATE function to specify the ASCII characters from blank (decimal ASCII code 32) to tilde (126) as the admissible characters. The subsetting IF statement excludes all observations where URL contains a character outside of this range.

Re: Detect or Exclude rows which include not standard ASCII characters

SASJedi — Fri, 28 Jul 2023 18:18:51 GMT

The COMPRESS function to the rescue! This example keeps only printable characters.

data want;
	set have;
	where compress(url,,'kw')=url;
run;

Re: Detect or Exclude rows which include not standard ASCII characters

Tom — Fri, 28 Jul 2023 20:15:44 GMT

That is going to include a LOT of non-ASCII characters.

91   data want;
92     url=collate(0,255);
93     expect=collate(32,126);
94     try=compress(url,,'kw');
95     if try ne expect then do;
96       extra=compress(try,expect);
97       put extra= / extra $hex. ;
98     end;
99   run;

extra=€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇ
8082838485868788898A8B8C8E9192939495969798999A9B9C9E9FA0A1A2A3A4A5A6A7A8A9AAABACAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7

Re: Detect or Exclude rows which include not standard ASCII characters

Patrick — Sat, 29 Jul 2023 00:10:43 GMT

Your sample data indicate otherwise but should you by any change be dealing with multibyte characters in your real data then none of the already proposed solutions would work and you need to look into SAS string functions on level I18N Level 2.
https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/nlsref/p1pca7vwjjwucin178l8qddjn0gi.htm

Re: Detect or Exclude rows which include not standard ASCII characters

Tom — Sat, 29 Jul 2023 05:14:10 GMT

Note that UTF-8 encoding is designed not to mess with normal ASCII codes, so the first (best) solution of using VERIFY() with COLLATE(32,126) will work fine on UTF-8 strings.

Re: Detect or Exclude rows which include not standard ASCII characters

Ksharp — Sat, 29 Jul 2023 09:35:11 GMT

data have;
length url $3000.;
input url;
datalines;
blogs.sas.com/wan/2022/03/18/sas-eg-è·³å‡ºéŒ¯èª¤è¨Š 
www.dog.it
;
run;

data want;
 set have;
 if prxmatch('/[[:^ascii:]]/',url) then flag=1;
run;