Undetermined I/O failure with shift-jis encoding, ja_JP locale

[JuanS_OCS]

Hello everyone,

this might be quite a basic question, actually I had in in the past, but I am not managing to recall completely or find the resolution, I hope you can help me with it.

Background:

I am am working in a global/multi-lingual environment. In SAS EG 7.15 HF8, under a SASApp server, if you extend the Files folder, then the home folder (or some other) that contains a file with double-byte characters (such as test_中國.sas), that folder won't open, or refresh. Instead, it gives a:

Undetermined I/O failure

ERROR SAS.EG.MainForm [(null)] - An exception was encountered while trying to retrieve the LastModified property for an element.
SAS.EG.SDS.SDSException: <?xml version="1.0" ?><Exceptions><Exception><SASMessage severity="Error">Undetermined I/O failure.</SASMessage></Exception></Exceptions> ---> System.Runtime.InteropServices.COMException: <?xml version="1.0" ?><Exceptions><Exception><SASMessage severity="Error">Undetermined I/O failure.</SASMessage></Exception></Exceptions>
   at SAS.IFileService_1_3.ListFiles_4(String Path, Int32 Mode, Array& FieldInclusionMask, String& ListedPath, Array& Names, Array& Typenames, Array& TypeCategories, Array& Sizes, Array& Modtimes, Array& Engines)
   at SAS.EG.SDS.Model.Util.ListFiles(IFileService fileService, String path, FileServiceListFilesMode mode, Array& fieldInclusionMask, String& listedPath, Array& names, Array& typeNames, Array& typeCategories, Int64[]& sizes, Array& modTimes, Array& engines)
   at SAS.EG.SDS.Model.SASFileService.Populate(String path, Collection& collection)
   --- End of inner exception stack trace ---
   at SAS.Shared.DNADecoder.CreateObject()
   at SAS.EG.MainForm.CheckForFileModification(FileChangeNotification fcn)

If I change this file's name to contain only single-byte characters, then the folder expands or refreshes OK.

Sometimes, not always, the same behavior it is seen in SAS Studio.

OK, so far, this is the common behaviour and expected, when you don't have the correct LOCALE/ENCODING pair in your SASApp server configuration. Now the fun part comes.

Current situation/what I know:

We have a total of 6 (six) SASApps, each of those with a different combination of env variables (the ones related to this question):

1-            latin1, en_US, ANY

2 and 3- shift-jis, ja_JP, V7

4-           utf8, en_US, ANY

5-           utf8, ja_JP, ANY

6-           wlatin1, en_US, ANY

Of course, the described behavior happens, and it is expected on 1 and 6 (I/O error), and in 4+5 (reading OK the files).

But I am very surprised that, in 2 and 3, being double-byte character sets and the right locale, they cannot read folders with those kind of filenames.

The Question, with capital Q:

I think all of you would agree with me that a SASApp having Japanese locale and encoding, should be able to read OK file names having Japanese characters.... so.... what is going wrong and how to make it right?

Thank you in advance for your comments and insights!

Best regards,

Juan

[ChrisHemedinger]

Hi @JuanS_OCS,

I don't think the locale setting is a factor here, just the encoding.  UTF-8 is the most "accepting" of course, as it should be able to represent all characters.  But shift-jis can be tricky sometimes, as it can represent native DBCS Japanese characters (as well as those you find in basic ASCII), but any characters that deviate from that can be a problem.

@Junichi might be able to say more about what's expected here.

[JuanS_OCS]

Hi @ChrisHemedinger , 

you are fast replying, or I had that good luck that you were looking at here when I posted the question.

I agree with you, in theory it should be related only to the encoding. When I dove deeper, I thought it could be a mix of encoding and locale since the deep error is related to lastModified.

Any help will be very welcome.

Thanks!

[JuanS_OCS]

Hi @ChrisHemedinger , @Junichi , all who  might be interested,

a kindred spirit from Tech Support has pointed me out that the issue comes because my WKS runs with Shift-JIS, reading filenames and directories from an OS  in a different encoding, UTF-8, and the fact that the Environment Variable PATHENCODING might help. Which I knew but completely forgot about it since I rarely use it!

I will give it a try and keep you posted, but I have my hopes up because this makes full sense.

Kudos to the answer from the Tech Support Team!