$ebcdic missing 8 bytes with locale off

[Tom]

It works because only the 16 characters 0-9 and A-F need to be translated from EBCDIC to ASCII in the transfer from the database.

[Tom]

If it is moving your data as binary then you should not have any issue with any binary fields in your data.  For the text fields that odds of a value being in that field that is not translated "properly" is much smaller and usually not that serious an issue.  In the old days we had issues with {} and [] and some other punctuation characters.  But since we didn't put non-printing characters into text fields there was normally no major impact from having the values go through an EBCDIC to ASCII tanslation.

If your binary data is being "translated" then as long as the conversion is not mapping more than one value to the same value. (ie mapping both 15x and 25x  to LineFeed) then you should be abel to roll you own conversion tool.  Put a test record into the database that is 256 characters long and has the values from 00x to FFx.  Pull it through the conversion and store it in a file.  Then use the translate function to convert the strings back to their original values.

Try this example.  Compare the value you get with the dataset below with the values that you receive from pulling the data out of the database to see where your process is different than the what $EBCDIC format assumed would have happened.

data canary ;

  converted=put(collate(0,256),$ebcdic256.);

  put converted $hex80.;

run;

data want ;

  if _n_=1 then set canary ;

  input string $20.;

  new = translate(string,converted,collate(0,256));

  put string $20. / string $hex40. / new $hex40. ;

cards;

1234567890ABCDEF

run;

1234567890ABCDEF

3132333435363738393041424344454620202020

F1F2F3F4F5F6F7F8F9F0C1C2C3C4C5C640404040

[Kurt_Bremser]

What is the reason that you don't use the s370fpd informat to read packed decimals?

[jakarman]

Greg, your refered to a note 14742 that advices to remove the locale setting .... that is another issue.

I give you the link to   MAPEBCDIC2ASCII=TRANTAB catalog-entry    (System Option) as it the one defining your own trantab. The associated note there:

Do not change a translation table unless you are familiar with its purpose.                                       

Translation tables are used internally by SAS to implement NLS. If you are unfamiliar with translation tables, do not change the specifications without proper technical advice.

For your fun I18N http://support.sas.com/resources/papers/proceedings13/025-2013.pdf

The systemprogrammer(s) on the mainframe is most likely to have set codepage 034 not 037 (z/OS and DB/2) . The same would be valid for your pc. US more likely 437, Europe as 850.

Using your ODBC-connection I only can see as something connecting to DB/2 it is not a filetransfer. Did you something special to have that in an binary connection? Normally that connection is included with that translation. This will spoil all your attempt on converting binary data to something usefull as you need to revert that ODBC translation (hidden in the driver). 

As mentioned it is very classic issue I have seen. The first reference System/370 gx20-1850-3 (1976).  Just looking at ISO/IEC 8859-1 - Wikipedia, the free encyclopedia (ascii) and notice the not being defined area's. That is different in the JAVA/Windows version.  

Seeing you reference to IBM-037 codepoints 4A and AC you see the mentioned chars refered to extended ascii points. The not sign is not part of normal ascii. That is why you language (including SAS) having more options to that. If your bypass on reversing the ODBC  Ebcdic/Ascii is working for you fine. I would not trust that as generic approach. 

[gsnidow]

Thank you Jaap for the link, some good stuff there.  I did use Trantab to update my syshelp.host translation tables, and when it did not solve my problem I googled some more and found, as you have indicated, they are used in file transfer, which I am not doing, so no help there.  I think I will just stick with my solution with the array, as it produces the correct result 100% of the time. 

[Kurt_Bremser]

It has been my experience that the S370/ebcdic formats behave differently over different SAS versions. One of our threads we opened with the SAS hotline even became a genuine error point in SAS 8.2 that lead to a fix.

I solved the problem by applying a macro to all strings read from the host that uses the translate() function to correct the Umlauts and various other characters. When we migrated to 9.2 (on AIX), I had to insert a check for the system version, because in 9.2 the $ebcdic format already gave us the correct characters.

[jakarman]

Let us keep it simple. When at the DB2 the records with their fields would have defined as separate fields being character and numeric ones, it was the ODBC / DB2 connection doing the translation.

As the problem is being introduced by referencing them as a combined field being first translated ebcdic-ascii it needed to be translated back into the original bytes.

[gsnidow]

Jaap, this is coming from one column in a DB2 table, not separate columns combined upon import.  I don't think anything is wrong with the odbc import, since I can, with 100% accuracy, turn the binary fields into the ebcdic hex they need to be, its just that I can't use a sas format to do it.  I have compared my ebcdic conversions to what I can get out of TSO on literally tens of millions of row without any errors.  I've been trying to get this to work for around 2 years now, and the only issue I can detect is the use of $ebcdic.  Anyhow, I sure appreciate all the help everyone has given me here.

Greg