topic Re: Hex Format in SAS Programming

Hex Format

David_Billa — Wed, 31 May 2023 18:23:29 GMT

I've used the format $hex50. to identify any special characters or invisible characters in the string. Value for the string is coming from .xls. Assume string value is "Urology". When I executed the program with $hex50. with the value "Urology", I got following message in the log. How to interpret this code to see if/what special characters in the string?

55726F6C6F67790D0A0D0A

Following code is removing the spaces between the strings. I want to remove special characters (spaces, CR, LF) before and after the string and NOT between the strings

compress(string,'090A0C0D20A0'x))

Re: Hex Format

Tom — Wed, 31 May 2023 15:59:43 GMT

Look up the ASCII codes. Or just play around yourself with the using the $HEX format on single characters.

You have :

The last four are two pairs of Carriage Return and Line Feed characters.

You probably do NOT want those in your actual text data. You could use COMPRESS() to remove them

string = compress(string,'0A0D'x);

Or perhaps use TRANSLATE() to replace them with spaces

string = translate(string,'  ','0A0D'x);

Re: Hex Format

David_Billa — Wed, 31 May 2023 17:25:03 GMT

@Tom Thank you for the information. I want to remove the special characters (spaces, CR, LF) before and after the string and NOT between the string. Currently following line is removing the spaces between the string which I don't want to do.

compress(string,'090A0C0D20A0'x))

E.g. If the string is 'Banking and Insurance' and I don't want to make it as 'BankingandInsurance' by removing the spaces between the string.

Re: Hex Format

David_Billa — Wed, 31 May 2023 18:26:06 GMT

I want to remove the special characters (spaces, CR, LF) before and after the string and NOT between the string. Currently following line is removing the spaces between the string which I don't want to do.

compress(string,'090A0C0D20A0'x))

E.g. If the string is 'Banking and Insurance' and I don't want to make it as 'BankingandInsurance' by removing the spaces between the string.

Re: Hex Format

Reeza — Wed, 31 May 2023 18:59:40 GMT

I believe 20 is the typical space characters. Why did you add all the whitespace characters to your remove list?

Re: Hex Format

David_Billa — Wed, 31 May 2023 19:02:11 GMT

I'm not certain how to remove white spaces and special characters before
and after the strings. Any help?

Re: Hex Format

Reeza — Wed, 31 May 2023 19:05:54 GMT

Assuming trim/strip have failed?

Re: Hex Format

David_Billa — Wed, 31 May 2023 19:11:11 GMT

Right, trim has failed and hence I moved to compress.

Re: Hex Format

Reeza — Wed, 31 May 2023 19:15:13 GMT

SAS automatically addes 20 (spaces to fill out a length of the variable) so that's fine, but otherwise, COMPRESS() seems to work.

 
data have;
input value $hex50.;
format value $hex50.;
cards;
55726F6C6F67790D0A0D0A
;
run;

proc print data=have;
run;
proc print data=have;
format value $50.;
run;

data want;
set have;
value2=strip(value);
value3=trim(value);
value4 = compress(value, '0D0A'x);

format value2 value3 value4 $hex50.;

run;

proc print data=want;
run;

Re: Hex Format

David_Billa — Wed, 31 May 2023 19:21:11 GMT

Thank you for the instruction. Whether 0D0A in compress function removes
special/invisible characters as well along with whitespaces before and
after the strings?

Re: Hex Format

Reeza — Wed, 31 May 2023 19:26:56 GMT

It does, you should only see 20 after the string in the variable value4. SAS always add extra spaces to pad out to the length of the variable from your example.

Re: Hex Format

Reeza — Wed, 31 May 2023 19:32:11 GMT

It doesn't remove every single special character - you need to explicitly list the ones you want removed. From your sample data it's the two shown.

Re: Hex Format

Tom — Wed, 31 May 2023 19:33:56 GMT

@David_Billa wrote:

I want to remove the special characters (spaces, CR, LF) before and after the string and NOT between the string. Currently following line is removing the spaces between the string which I don't want to do.
compress(string,'090A0C0D20A0'x))
E.g. If the string is 'Banking and Insurance' and I don't want to make it as 'BankingandInsurance' by removing the spaces between the string.

If you don't want to remove the spaces then take the 20 out of your COMPRESS() function all.

Usually I use translate to replace the goofy characters with spaces and use COMPBL() to reduce them to one per break. You can use LEFT() to remove the leading spaces.

left(compbl(translate(string,' ','090A0C0DA0'x)))

No need to use TRIM() , or STRIP(), to remove the trialing spaces because they will just get added back when you store the result into a variable.

Re: Hex Format

David_Billa — Wed, 31 May 2023 19:38:11 GMT

I want to remove all the special/invisible characters including spaces
before and after the string and NOT between the string.

Re: Hex Format

Reeza — Wed, 31 May 2023 19:42:32 GMT

What's a special character? Hyphen? Exclamation mark? This is why you need to list the characters of interest.

Re: Hex Format

Tom — Wed, 31 May 2023 21:58:06 GMT

@David_Billa wrote:
I want to remove all the special/invisible characters including spaces
before and after the string and NOT between the string.

If you want to use COMPRESS() to remove characters you have to tell it exactly what characters to remove. So if you don't want it to remove spaces then don't tell it to remove them.

If it is easier you can use the K modifier and instead tell it which characters to keep. For example if you only want to keep the normal ASCII characters just use something like:

new_string = compress(old_string,collate(32,126),'k');

32 is 20 in hex which is a space. 126 is 7E in hex which is a ~. All of the chracters from space to ~ are in printable standard 7-bit ASCII characters. Everything else is either a control character like CR or LF or some strange non-ASCII character like accented letters or strange symbols.

If you are having trouble figuring out what characters you have that is what the $HEX format can help with.

The STRIP() function will remove leading and trailing SPACES. It does not remove any other characters. As I explained before there is no need to remove the trailing spaces since they just come back when you assign the value to a variable. SAS uses FIXED LENGTH character variables. So they are always padded with spaces to fill the full storage length. So LEFT() is enough to remove the leading spaces.

Not sure what you mean by "between strings". To you mean between the words in the current character variable? That is why I suggested using the COMPBL() function. That will collapse multiple adjacent spaces into one space, but does not remove the space between the words.

Or are you planning to combine a series of character variables together? If so then explain how you are going to combine them to help on preventing extra spaces being inserted.

Re: Hex Format

Kurt_Bremser — Thu, 01 Jun 2023 06:17:17 GMT

Do not use "20"x in the COMPRESS(); remove leading and trailing blanks with STRIP().

Re: Hex Format

yabwon — Thu, 01 Jun 2023 09:25:00 GMT

If you want to delete leading and trailing spaces('20'x) and "enters" (LF, CR) you could try to "resolve" it his way

data test;
  x='0A20200D55722055720D0A6F6C6F67790D0A20200D0A0D0A0D0A0D0A'x;
  put "*" x= char. +(-1)'*';
  put "*" x= hex. +(-1)'*';

  s=FINDC(x,, 1,'KS'); /* first "non space" */
  e=FINDC(x,,-vlength(x),'KS'); /* last "non space" */
  put s= e=;

  x=substr(x,s,e-s);
  put "*" x= char. +(-1)'*';
  put "*" x= hex. +(-1)'*';
run;

Bart

Re: Hex Format

David_Billa — Thu, 01 Jun 2023 10:10:45 GMT

@yabwon I want to delete CR, LF and Spaces or any special characters before and after the string. Then how to tweak your code?

Re: Hex Format

yabwon — Thu, 01 Jun 2023 10:17:49 GMT

Define "special characters" ?

The answer will probably be: change FINDC() functions modifiers.

Basically "SK" means "search for first character that is NOT a space, a horizontal tab, a vertical tab, a carriage return, a line feed, and a form feed".

Probably you would like to add "C" for control characters, so in total "SCK".

Or maybe just make it: "ANP" to search for first alphabetic character, punctuation mark, or digits, an underscore, and English letter?

Bart

P.S. are you using two accounts: @David_Billa and @Babloo ?