topic Re: Solaris | Encoding in SAS Programming

Solaris | Encoding

MariaD — Wed, 15 Jul 2020 15:12:18 GMT

Hi folks,

We are using a new SAS environment on Solaris. Our users have some folders and tables name with latin characters (for example: "ó").

Under PuTTY we verify the folder exists and is as expected: "/usuarios/nuevo_usuario/Histórico"

On SAS, using SAS EG 7.15, we define a libname with the mentioned path. According log information, the libname was correctly assigned to the defined path.

26         libname test '/usuarios/nuevo_usuario/Histórico';
NOTE: Libref TEST was successfully assigned as follows: 
      Engine:        V9 
      Physical Name: /usuarios/nuevo_usuario/Histórico

28         data test.sample;
29           set sashelp.class;
30         run;

NOTE: There were 19 observations read from the data set SASHELP.CLASS.
NOTE: The data set TEST.SAMPLE has 19 observations and 5 variables.
NOTE: Compressing data set TEST.SAMPLE increased size by 100.00 percent. 
      Compressed is 2 pages; un-compressed would require 1 pages.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.00 seconds

We execute a simple data step to create a table. According the log, again, everything is fine. But verifying the server using PuTTY, we realise the table as created inside a different path: /usuarios/nuevo_usuario/HistÃ³rico

As you can see, the "ó" character was converter from Latin to UTF. There is any way to prevent it? Any SAS configuration or Solaris configuration?

Regards,

Re: Solaris | Encoding

LeonidBatkhan — Wed, 15 Jul 2020 19:14:39 GMT

Hi MariaD,

Check your sasv9.cfg file ( located somewhere at ...SASHome/SASFoundation/9.4/nls/en ) and make sure it has the following encoding set:

-ENCODING wlatin1

Hope this helps.

Re: Solaris | Encoding

MariaD — Wed, 15 Jul 2020 19:44:24 GMT

Thanks @LeonidBatkhan , I'll try it and let you know. I need to restart SAS services after the change? I believe that it's not necessary.

Re: Solaris | Encoding

ChrisNZ — Thu, 16 Jul 2020 01:41:32 GMT

So you see 2 folders on the server? Histórico and HistÃ³rico?

When is the second one created? After the libname statement is run?

Re: Solaris | Encoding

Kurt_Bremser — Thu, 16 Jul 2020 09:11:50 GMT

Make it a habit (and beat this into the heads of all your users) that you NEVER use anything other than

the 26 letters of the standard alphabet
the 10 digits of the decimal system
underlines

in directory and file names on servers. NEVER. It only causes problems and serves no real purpose.

Client settings can cause all kinds of problems when trying to access files with UTF characters in the name.

Anecdote:

My MacBook failed to read mp3 files on my family NAS because an idiot system (Windows) wrote the single quotes in some titles as UTF single quotes, which appeared as funny character sequences in the Finder. Probably because the NAS coughed up on it.

Re: Solaris | Encoding

ChrisNZ — Thu, 16 Jul 2020 21:43:27 GMT

@MariaD

I agree with @Kurt_Bremser: IT is still a mess. Just make life easier for yourself by avoiding silly issues.

And I should add that his native language (like mine) uses "accented" letters too, so his reply is not born out of intolerance for funny squiggles.

Re: Solaris | Encoding

Kurt_Bremser — Thu, 16 Jul 2020 12:27:15 GMT

As @ChrisNZ noted, my native language is German, and we do have the umlauts (Ä,Ö,Ü) and the so-called "sharp s" ß. Before those letters even appeared on mechanical devices (like IBM typewriters with German heads), it was common to translate those to ae, oe, ue and ss, respectively. When I started in IT, no computer knew those letters, so the translation was even used in official documents, if those were printed by computers.

Nowadays, users are tempted to use these characters in filenames or elsewhere in a technical context.

But see:

The problem of native language characters was approached in stages; first, by using some bytes in the ASCII table for several purposes in different parts of the world (i.e., square brackets were replaced by Ä and Ü in certain German codepages, as square brackets are rarely used in German). This was done to keep using only 7 bits for a character (look into manuals of older terminals to understand this). Next, the "upper half" (most significant bit = 1) of the ASCII table was used, which can be found in the codepages of UNIX systems and Windows, which accommodated most of the Western European orthography. But with the need to use completely different writing systems (Cyrillic, Arabic, Hebraic, Korean, Japanese, Chinese), a much larger table of characters/symbols was implemented with the UTF system. The problem now is that the UTF characters need "entry points" from the single-byte character table, so they be identified as UTF; but the bit sequences of those "entry points" are already used in certain codepages for displayable characters, and if a data stream is not explicitly marked as UTF, lots of systems misunderstand the two-, three- or four-byte sequence of a UTF character for the usual characters associated with these 2/3/4 codes, resulting in hilarious (or not so) hilarious results.

While desktop systems have learned to deal with this, there are areas in computing where UTF use can have serious consequences. Consider the fact that operating systems (including Windows!) have a hard limit of ~200 characters (bytes) for path names. A fully Chinese filename would only need 50 or so UTF characters/symbols to exceed that.

There are other problems: as I experienced, there are now (at least) two single quotes available: the standard one, as also used in code, and a UTF one. That caused the "fun" with my MP3 files. Text writing software will auto-replace characters (Word, in German, will auto-replace double quotes with the "leading" and "trailing" quotes used in German, or it will use "beautified" (slightly curly) quotes; copy/pasting code with those will inevitably cause syntax errors).

Another serious issue that happened to me: I used the letter "Ä" in a password, to follow the "at least one special character" rule. One computer sent the character from the Wlatin codepage, the other the UTF equivalent. Consequence: a locked user.

Bottom line: on the technical side (names in code, filenames, server names, user ID), stay with the basic English-derived lettering system, and follow my previous rules; especially, avoid the use of blanks, and replace those with underlines to separate words in file or variable names. Put fancy text into labels. A label might display "funny" on a remote system, but it won't crash your program, or cause files to be un-openable.

If you follow these rules, somebody from the other side of the world will always be able to help you with technical issues over a remote access connection.

Re: Solaris | Encoding

MariaD — Thu, 16 Jul 2020 12:41:48 GMT

Hi @ChrisNZ ,

The first one was created through SAS Enterprise Guide, under Files Section (New Folder and then renamed to "Histórico"). After we created through SAS Enterprise Guide, we validate using PuTTY and we discovered that the folder created appears as "HistÃ³rico". Using the same user, but through PuTTY interface, we create a new folder using mkdir command, in this case the folder created appears as " Histórico".

Regards,

Re: Solaris | Encoding

MariaD — Thu, 16 Jul 2020 12:42:55 GMT

Hi @LeonidBatkhan ,

On sasv9.cfg ENCODING settings is already as LATIN1.

Regards,

Re: Solaris | Encoding

MariaD — Thu, 16 Jul 2020 12:46:25 GMT

Hi @Kurt_Bremser ,

Yes, you are correct but some business user uses latin characters on folders and files, even when we tried to suggest to not to use. Our previous SAS server was Linux, so we have a lot of files and folders with this situation.

Regards,

Re: Solaris | Encoding

Kurt_Bremser — Thu, 16 Jul 2020 13:30:56 GMT

The change needs to be made in sasconf/Lev1/SASApp/sasv9_usermods.cfg, after which your SAS services need to be restarted.

Re: Solaris | Encoding

ChrisNZ — Thu, 16 Jul 2020 22:18:50 GMT

@Kurt_Bremser is so much more dedicated than I am!

My "IT is still a mess" looks lazy now, compared to his detailed comments! 🙂

Bottom line: Different systems use different character representations, and until all systems use the same one (UTF-8 seems to be becoming the new standard, but Windows is very late to the party), this issue is unavoidable.

One system writes something using encoding A, another system expects encoding B, and wheels come to a stop.

Maybe in 20 years all systems and all legacy data will be UTF-8 (here is wishful thinking), and we'll breathe easier.

Two thoughts:

1. As far as I know, Solaris's default encoding is UTF-8. Why do you have latin folder names? I wonder if Putty is confusing things here. What do you use it for?

2. Run proc options group=languagecontrol; run; to see what encoding is set for the SAS session.

Re: Solaris | Encoding

MariaD — Fri, 17 Jul 2020 17:28:23 GMT

Hi @ChrisNZ ,

Follows the results of proc options:

Group=LANGUAGECONTROL
 DATESTYLE=MDY     Specifies the sequence of month, day, and year when ANYDTDTE, ANYDTDTM, or ANYDTTME informat data is ambiguous.
 DFLANG=ENGLISH    Specifies the language for international date informats and formats.
 DSCAS             Runs the DATA step on the CAS server.
 EXTENDOBSCOUNTER=YES
                   Specifies whether to extend the maximum number of observations in a new SAS data file.
 LOCALEDATA=SASLOCALE
                   Specifies the location of the locale database.
 NOLOGLANGCHG      Disables changing the language of the SAS log when the LOCALE= option is changed.
 NOLOGLANGENG      Write SAS log messages based on the values of the LOGLANGCHG, LSWLANG=, and LOCALE= options when SAS started.
 LSWLANG=LOCALE    Specifies the language for SAS log and ODS messages when the LOCALE= option is set after SAS starts.
 MAPEBCDICTOASCII= Specifies the transcoding table that is used to convert characters from ASCII to EBCDIC and EBCDIC to ASCII.
 NONLDECSEPARATOR  Disables formatting of numeric output using the decimal separator for the locale.
 NOODSLANGCHG      Disables changing the language of the SAS message text in ODS output when the LOCALE option is set after start 
                   up.
 PAPERSIZE=LETTER  Specifies the paper size to use for printing.
 RSASIOTRANSERROR  Displays a transcoding error when illegal values are read from a remote application.
 TIMEZONE="GMT-03:00"
                   Specifies a time zone.
 TRANTAB=(lat1wlt1,wlt1lat1,lat1_ucs,lat1_lcs,lat1_ccl,,,)
                   Specifies the translation table catalog entries.
 URLENCODING=SESSION
                   Specifies whether the argument to the URLENCODE function and to the URLDECODE function is interpreted using the 
                   SAS session encoding or UTF-8 encoding.
 NODBCS            Disables double-byte character sets.
 DBCSLANG=NONE     Specifies a double-byte character set language.
 DBCSTYPE=NONE     Specifies the encoding method that is used for a double-byte character set.
2                                                          The SAS System                              17:48 Thursday, July 16, 2020

 FSDBTYPE=DEFAULT  Specifies a full-screen double-byte character set (DBCS) encoding method.
 FSIMM=            Specifies input method modules (IMMs) for full-screen double-byte character sets (DBCS).
 FSIMMOPT=         Specifies options for input method modules (IMMs) that are used with a full-screen double-byte character set 
                   (DBCS).
 ENCODING=LATIN1   Specifies the default character-set encoding for the SAS session.
 LOCALE=EN_US      Specifies a set of attributes in a SAS session that reflect the language, local conventions, and culture for a 
                   geographical region.
 NONLSCOMPATMODE   Encodes data using the SAS session encoding.

Thanks,

Re: Solaris | Encoding

MariaD — Fri, 17 Jul 2020 17:33:08 GMT

An additional comment, the results of PROC OPTIONS for Linux (on Linux works fine) environment is exactly the same.

Regards,

Re: Solaris | Encoding

Tom — Fri, 17 Jul 2020 19:13:25 GMT

@MariaD wrote:

Hi @ChrisNZ ,

The first one was created through SAS Enterprise Guide, under Files Section (New Folder and then renamed to "Histórico"). After we created through SAS Enterprise Guide, we validate using PuTTY and we discovered that the folder created appears as "HistÃ³rico". Using the same user, but through PuTTY interface, we create a new folder using mkdir command, in this case the folder created appears as " Histórico".

Regards,

By PUTTY do you mean that you used that software to open a terminal window on your unix system and typed the unix command mkdir at the command prompt? What encoding was your terminal window using? What happens if you use SAS command to make the directory?

data _null_;
  infile 'mkdir /path with non 7bit ascii letters' pipe;
  input;
  put _infile_;
run;

Re: Solaris | Encoding

MariaD — Fri, 17 Jul 2020 20:32:14 GMT

Hi @Tom ,

Follows the results:

NOTE: The infile 'mkdir /path/test' is:
      Pipe command="mkdir /path/test"

NOTE: 0 records were read from the infile 'mkdir /path/test'.
NOTE: DATA statement used (Total process time):
      real time           0.08 seconds
      cpu time            0.01 seconds

By terminal, such PuTTY or command, folder or files with special characters are created find and we can navigate through it.

One more point, we can't see SAS tables with special characters on name under LIBNAME. We know this is not the best practices, but sometimes our business users create this type of file too.

Regards,

Re: Solaris | Encoding

ChrisNZ — Sat, 18 Jul 2020 00:20:50 GMT

Here is my best guess, a stab in the dark really.

Findings & facts so far (you haven't replied to some questions):

- Solaris should use UTF-8

- Putty should use UTF-8

- The folder name is displayed correctly in Putty, which makes sense as they use the same encoding: The folder name uses UTF-8 too.

- SAS uses wlatin1 for its data

- Because SAS is aware of the underlying OS, is reads the folder name correctly, using UTF-8 (1)

- A new folder is created by SAS (please confirm you have 2 folders) when you run SAS code (2)

- This new folder has a name that is not interpreted as UTF-8 by Solaris and Putty for some reason. We don't know why (3).

=> There are still many gaps in our establishing the facts.

One logical answer would be to set SAS to use UTF-8 so everything is coherent.

UTF-8 seems to be the new emerging standard anyway, so moving there is desirable.

Can you try that to see if it fixes this one issue?

Note that some data might need translating. SAS can do that on the fly using CEDA.

Notes:

(1) The folder name is displayed correctly:

26         libname test '/usuarios/nuevo_usuario/Histórico';
NOTE: Libref TEST was successfully assigned as follows: 
      Engine:        V9 
      Physical Name: /usuarios/nuevo_usuario/Histórico

(2) I don't see why a new folder should ever be created when running a data step.

The creation may be linked to the LIBNAME statement instead, provided option DLCREATEDIR is turned on.

Please confirm when the folder is created.

(3) You could look at the folder name using an hex editor to see each individual byte, and see why a seemingly legal UTF-8 byte sequence is not combined.

Re: Solaris | Encoding

MariaD — Mon, 20 Jul 2020 12:48:11 GMT

Hi @ChrisNZ ,

Please find my comments below:

- Solaris should use UTF-8 -- It's UTF 8

- Putty should use UTF-8 -- It's UTF 8

- SAS uses wlatin1 for its data -- Yes, on our previous environment (Linux) was already setting as WLATIN1

- A new folder is created by SAS (please confirm you have 2 folders) when you run SAS code (2) -- No, the folder is not created when run the libname statement. One of the folders, the corrected one, was created using mkdir through PuTTY. The other one, was created through SAS EG under Server --> Files --> New Folder. The last one, appears as " HistÃ³rico".

- This new folder has a name that is not interpreted as UTF-8 by Solaris and Putty for some reason. We don't know why (3). -- In fact, SAS does not interpret the corrected name, "Histórico", for folders and for SAS datasets too. On Solaris or PuTTY, we can navigate through folders without any problem.

Can you try that to see if it fixes this one issue? -- I'll make a test. But in our previous environment (Linux) we have exactly the same configuration and works fine on it.

Note that some data might need translating. SAS can do that on the fly using CEDA. -- Yes, we use a CPORT procedure to translate from Linux to Solaris.

Follows an example to visualise what is going on.

Case 1:

a) I created a folder called "Histórico" through PuTTY.

b) I'll try to assign a libname TEST using the folder "Histórico" on SAS EG.

25         GOPTIONS ACCESSIBLE;
26         libname TEST '/usaurios/sasdemo/Histórico';
NOTE: Library TEST does not exist.

Case 2

a) I deleted the previous folder created. I created again a folder called "Histórico" but now through SAS EG (Server --> Files --> New Folder). If I navigate to the folder using PuTTY, now the folders appears as "HistÃ³rico"

b) I run the same libname statement and now the libname is assigned.

25         GOPTIONS ACCESSIBLE;
26         libname TEST '/usuarios/sasdemo/Histórico';
NOTE: Libref TEST was successfully assigned as follows: 
      Engine:        V9 
      Physical Name: /export/home/sasdemo/Histórico

Regards,

Re: Solaris | Encoding

Kurt_Bremser — Mon, 20 Jul 2020 14:02:12 GMT

What happens if you use the DCREATE() function?

Is the directory accessible through the LIBNAME statement?

How is it displayed in putty?

From your description, I now have the suspicion that EG and SAS use UTF-8 encoding when sending the name to the filesystem, but putty recognizes the two-byte UTF sequence as two separate WLATIN characters.

Do you have legacy directories that were created on the Linux system and then either mounted to your Solaris server, or copied through NFS? If yes, how do those display in EG and putty?

How did you determine that the Solaris server is configured to use UTF?

Re: Solaris | Encoding

Tom — Mon, 20 Jul 2020 14:55:39 GMT

First step is to run SAS using UTF-8 encoding. This has to be set when the SAS session starts, you cannot change it later.

Then try making a file or directory using some non-7 bit ASCII character, like your example o with accent.

Then try seeing what the name looks like in the filesystem. For example you might try reading the output of the LS command.

data test;
  infile
"cd /usuarios/nuevo_usuario/
;ls -d His*
"  pipe truncover ;
   input filename $30. ;
   put _n_= filename=  +1 filename $hex60. ;
run;

Now you should be able to see the names of the two different His... directories and the actual character codes.

Here is my attempt to see the difference in what bytes that name has when using UTF-8 vs LATIN encoding by running the same data step in two different SAS sessions.

Notice how just pasting the UTF-8 text into this forum causes the UTF-8 character to be interpreted as two independent non-7bit ASCII characters.

22    data _null_;
23      length encoding name $20;
24      encoding=getoption('encoding');
25      name = 'HistÃ³rico';
26      len=length(name);
27      klen=klength(name);
28      put (_all_) (=) / name $hex. ;
29    run;

encoding=UTF-8 name=HistÃ³rico len=10 klen=9
48697374C3B37269636F20202020202020202020

1317  data _null_;
1318    length encoding name $20;
1319    encoding=getoption('encoding');
1320    name = 'Histórico';
1321    len=length(name);
1322    klen=klength(name);
1323    put (_all_) (=) / name $hex. ;
1324  run;

encoding=WLATIN1 name=Histórico len=9 klen=9
48697374F37269636F2020202020202020202020

So when using UTF-8 the character is coded as C3 B3 and when running WLATIN1 the character is coded as F3.