topic Re: SAS UTF- 8 Unicode in SAS Programming

SAS UTF- 8 Unicode

neethubabu — Tue, 01 Jun 2021 17:45:02 GMT

Hi, I am getting the below error when I am working on the SAS dataset. I think this is related to SAS UTF- 8 Unicode. Is there any way to deal with this? I need to do the same in multiple SAS datasets. Can anyone help me find a solution?

Re: SAS UTF- 8 Unicode

MarkBodt_NZ — Wed, 02 Jun 2021 03:15:20 GMT

Hi,

This is a common issue when using data that was created on one server on a different server. Often experienced when migrating from an older SAS environment to a new one, or a Windows server to a Unix server

Have a read of the documentation here:

https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/movefile/n06qan4j3ffr6fn11bs4q11r8r56.htm

There's a section that discusses your error along with solutions.

You mention that you need to convert multiple datasets. I can't offer you code solutions, but I can reflect on how I went about a similar issue at a site where I carried out a migration:

They had I think about 100,000 or so datasets that needed converting. The datasets were strewn through out many directories, I think about 30,000 directories if I remember corrrectly.

The approach I took was that I wrote a SAS program that 'spidered' a data volume, that is, it examined each directory, starting at the highest level and then each subdirectory recursively. It examined every SAS dataset as well as other SAS artifacts such as catalogs to see if they needed to be converted to UTF-8. It also checked for things like password protection, corrupt datasets etc.

Anyway, out of this, it produced a big 'shopping list' of SAS datasets that needed conversion, along with the path name

Part two of the process was a SAS program that used the 'shopping list' to assign libnames and execute code to read the dataset and convert it to UTF8. It did this using methods discussed in the documentation link above.

I hope that's helpful information.

Re: SAS UTF- 8 Unicode

ChrisNZ — Wed, 02 Jun 2021 03:38:08 GMT

If your title is correct, it looks like you are reading UTF-8 data in a session that does not support this encoding (probably some form of wlatin).

UTF-8 can encode many characters that do not exist in wlatin. So these characters are lost, as the message describes.

To avoid losing them, you should run your SAS session using UTF-8 encoding. That's a start-up option.

A similar issue arises when moving files to/from mainframes as some characters only exist in ASCII or EBCDIC representation.