topic Re: How to treat sas7bdat file with invalid characters? in SAS Programming

How to treat sas7bdat file with invalid characters?

Nietzsche — Sat, 22 Oct 2022 06:23:57 GMT

One of the data files storm_2017.sas7bdat that came with programming 2 has lots invalid characters causing errors.

what is the best way to treat these invalid characters? I am thinking about downloading it as CSV and removing them manually and then uploading up back into SAS.

Re: How to treat sas7bdat file with invalid characters?

Kurt_Bremser — Sat, 22 Oct 2022 07:06:23 GMT

Make a dictionary of characters and their UTF replacements (e.g. in an array), and then use TRANSTRN to do each replacement.

Re: How to treat sas7bdat file with invalid characters?

Nietzsche — Sat, 22 Oct 2022 08:22:33 GMT

I am sorry I am still a beginner, I kinda understand what you're saying but I have no idea how to implement it in SAS.

Re: How to treat sas7bdat file with invalid characters?

Ksharp — Sat, 22 Oct 2022 09:47:05 GMT

It looks like a carriage return character '\n' .
Can you change your sas 's encoding into 'utf-8' or others ? maybe could solve it .

Re: How to treat sas7bdat file with invalid characters?

Kurt_Bremser — Sat, 22 Oct 2022 20:29:09 GMT

This will take some research on your part. Use the $HEX format to display the hex codes of your strings, so you can use them for defining your "target" characters (see the function documentation).

From the context, you will be able to determine the wanted UTF character (in "Yucatan", it must be some kind of accented a). Look up the multi-byte "replacement" string in online available UTF tables.

Edit: fixed typos

Re: How to treat sas7bdat file with invalid characters?

Tom — Sat, 22 Oct 2022 12:47:08 GMT

Does it really cause any trouble for doing the exercises? If not then just ignore it.

To figure out what is going on do a little research.

Find out what encoding your SAS session is using.

proc options option=encoding;
run;

Find out what encoding the dataset is using.

proc contents data=mylib.storm_2017;
run;

There are two main types of encodings used.

Single byte encodings such as LATIN1 or WLATIN1. In those encoding each character uses only one byte. But that means that there are only 256 possible characters (which includes invisible characters like TAB, CARRIAGE RETURN and LINE FEED).

And multibyte encoding, which will normally be UTF-8. With UTF-8 some characters require 2,3 to 4 bytes to be represented.

So there literally thousands of characters that can be represented.

If your SAS session is using a single byte encoding then it might be that the characters in the dataset are ones that are not included in the 256 characters available in that encoding. So if your SAS session is NOT using UTF-8 as the encoding then see if you can change that. That way SAS will be able to represent any character that might be in the dataset.

Note: If the encoding settings on the dataset are confused you could still have invalid strings. For example if the strings in the dataset are actually using a single byte encoding, but SAS thinks they are UTF-8 then there are some combination of bytes that are not valid in UTF-8. That can also be fixed, but might require more effort.

Re: How to treat sas7bdat file with invalid characters?

Nietzsche — Sat, 22 Oct 2022 20:15:20 GMT

yeah, but I just completed the exercise with another sas7.dbat file