Your SAS programs, embedded in web apps and elsewhere

Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Accepted Solution Solved
Reply
Contributor
Posts: 28
Accepted Solution

Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

I have a big problem.When I use SAS EG to finish my model,there is always a mistaken like the picture.

How can I solve it?

Please give some details about that.thank you so much.IMG_1740.JPG


Accepted Solutions
Solution
‎09-12-2014 02:58 AM
Valued Guide
Posts: 3,208

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

gingyu, It is amazing how easy and far you are doing with this not easy SAS challenges. Not being mindblocked, solving each topic one by one in multitasking mode.

That  /tmp  location is the default for a Unix temporary files  (Posix Linux apology as I use the old AT name) . I do not kow how many users you have on the system and what your options are for file-systems and storage.  You could extend the /tmp filesystem but as it is the common for all tmp files on the machines may not the best way. You could set up some new mountpoints like:

/var/local/sas/tmpusr        -> for personal users the workspace server appserver

/var/local/sas/tmpbtch      -> for shared operations batch processing of an it-operations like

                                         SAS(R) 9.4 System Options: Reference, Third Edition (work)

/var/local/sas/tmputl         -> for the interenal tmporary files of Proc sql, proc sort. These filesnames are named  #UTL.... normally invisible

                                         SAS(R) 9.4 System Options: Reference, Third Edition (utillloc)
You do not possible need all of them it is only for loadbalancing having multiple users/processes running on the system

Every mountpoint/filesystem can be setup with enough space (Unix administration) for your intended processing. Any idea of your biggest datset sizing you will proces? make it at least 4 times bigger than that.  (I forgot) set the acces rights on those directory like /tmp. A chmod of 3777 or 3770 will do setting the sticky bit and groupid bit. If a group access can be used to limit others can left out.

Check those filesystems when running out of space. Regular Cleanup of workfiles can be automated (using cron) later, something to park in your mind now.

Having those filesystems in your unix you can point to them using the sas-config-files.  (Lev-/SASApp\sasv9_usermods.cfg). dedicated nested one can be used to give a SP or batch-server an other work-location

-work /var/local/sas/tmpusr

-utilloc /var/local/sas/tmputl  

---->-- ja karman --<-----

View solution in original post


All Replies
Valued Guide
Posts: 3,208

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Have your SAS session run a utf-8 one.  You will better use several connections (possible with server side) with utf8/latin-1 encodings.

Using a utf-8 sas session will require attention to character functions as not all of them are valid with utf8.

SAS(R) 9.4 National Language Support (NLS): Reference Guide, Third Edition


Would go for 9.4 with SAS/connect better supporting the interconnections.

---->-- ja karman --<-----
Contributor
Posts: 28

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

I appreciate your help! I realized the mistaken was something wrong with encoding.But how can I have my SAS session run a utf-8 one ? The encoding of my SAS server side is latin1 default.How can I change the configuration about encoding?

Respected Advisor
Posts: 3,892

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Contributor
Posts: 28

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

I found a solution on internet.The solution says adding the latin1 encoding to sasv9_usermods_cfg file.But I don't know how to do that. 

Respected Advisor
Posts: 3,892

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

You need to have admin privileges to change the .cfg. But have a look at the links I've posted. Some of the options work also as part of an options statement.

This bit in the docu for an option tells you where you can set/change it. If it says "configuration file, SAS invocation" then it's the .cfg or the autoexec, if it's "OPTIONS statement" then you can also set it in your code.

Valid in:configuration file, SAS invocation, OPTIONS statement, SAS System Options window
Contributor
Posts: 28

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

I reset the server side and deployed it as a unicode server.The error was solved.Maybe your answer also can solve this error,But the time is not enough for me.Thank you so much!

I also want to ask one question.

Now,I have a client,a sas application server and a database server(DB2),They are located in different computers.

I can use 'libname' statement to connect to the database.Can I directly analyse the data on the logic table from database?(the error :failed to I/O )

Should I move the data to the sas application server and then to analyse?

Valued Guide
Posts: 3,208

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Needed to process your DB2 question. When you are connecting that by a libname (db2 client configured and sas/access DB2)  working,  you can use that as input for analyses. just take care for the datatypes: SAS/ACCESS(R) 9.4 for Relational Databases: Reference, Fifth Edition

Wanting to do more with that interface you could use DB2 specific functions with explicit pass through or going for DS2.
DS2 is new world for accessing modifying data. SAS(R) 9.4 DS2 Language Reference, Third Edition (if you have time left to spend)    

---->-- ja karman --<-----
Contributor
Posts: 28

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

I have successfully connected the database using 'libname' statement like following picture(can see the tables):

QQ图片20140911162908.jpg

then there are two errors when I use this code;

QQ图片20140911163413.jpg

QQ图片20140911162158.png

this is my database server code set(gbk):

QQ图片20140911162240.png

So the reason why cause errors is the encoding of database is gbk and the application server is unicode?

Respected Advisor
Posts: 3,892

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

As points out you're not having enough disk space in your SAS work area.

To answer one of your other questions: Yes, it's a good idea to push processing to the data base. The normal approach is to first reduce volumes on the data base and then only move the data you actually need over the network into SAS.

For your code: If you can, select only the columns you need (so no Select *) and subset the rows if possible (where ....). Also should you only need summary values then push this bit as well to the data base (eg. using a group by statement in your SQL code).

Some SAS procedures allow you to push all processing to the data base (eg. Proc Freq and Proc Means). So if you can get your desired result with these procedures then use them.

Using below options shows you in the log where processing happens  SAS/ACCESS(R) 9.4 for Relational Databases: Reference, Fifth Edition

options sastrace=',,,d' sastraceloc=saslog nostsuffix;

There are also other techniques for improving performance when dealing with data stored in a data base. If you're working regularly with DB2 then it might be worth to spend some time and get up-to-speed with these techniques.

Here only one of many possible links: http://support.sas.com/documentation/cdl/en/acreldb/67473/HTML/default/viewer.htm#n13m5rcjajo3c5n12l...

Contributor
Posts: 28

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Thanks a lot.I learn much about SAS this time.

I hope you and  can help me when I face new problems.:smileylaugh:

Valued Guide
Posts: 3,208

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Multilanguage is becoming better.  http://support.sas.com/resources/papers/Multilingual_Computing_with_SAS_94.pdf

Still you have to deal with an en (English latin1 single byte) and u8 (utf8 multibyte) sas session.  To avoid compatiblity issue uses several appservers

http://support.sas.com/resources/papers/AddingAdditionalSASWorkspaceServerstoSupportMultipleEncoding...

Having done a serverside installation supporting Unicode and en language it is small configuration step to add an addtional server and change the sas-scriptname that will be used for the app-server (u8 or en) in the usermods-files.

Before 9.4 connections of sessions with different encoding is a disaster SAS/CONNECT(R) 9.2 User's Guide (Connect) and Proc cport / cimport also fails.

As you are mentioning DB2, that one is often limited to latin-1. When that is the case the DB2-client is ok.

Going for  DBMS supporting Unicode is posing an other issue. The RDBMS client will possible give back 4 bytes when in utf8 mode. With latin1 you need just one. As the DBMS client does this translation you possible need two configurations of the DBMS client.
Check this when you are getting into unexpected growth of char variables. (multiplied by 4). 
  

---->-- ja karman --<-----
Contributor
Posts: 28

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

It's me again.I have a new problem.

I want to test the highest pressure of my database server and SAS server(the maximum of SAS users) .

I have no idea about that.

Could you give me some advice or methods?

Valued Guide
Posts: 3,208

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

You have a working connection nice.
It wil get the data to the sas-dataset work.test , then putting data there coming from db2 is running out of space of the saswork
SASwork is a standard name in SAS for temporary data. There is an other utilloc that is special for utilities  sort sql.

You analyses will probably run with small data (eg obs100).

I do not know how you configured that work, but it is running out of space.

Can you check the physical name and sizing on the server of that one?  To get the information on that:  

libname work list ;  /* shows all attributes */

proc options; run;  /* shows all std options   look for work   */

---->-- ja karman --<-----
Contributor
Posts: 28

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

yes.I checked the information about my SASWork.

QQ图片20140912101529.png

it's too small to copy the whole data from database.How to enlarge the size of SASWork?

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 25 replies
  • 6368 views
  • 6 likes
  • 3 in conversation