topic Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE in Developers

Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Wed, 10 Sep 2014 03:35:25 GMT

I have a big problem.When I use SAS EG to finish my model,there is always a mistaken like the picture.

How can I solve it?

Please give some details about that.thank you so much.

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jakarman — Wed, 10 Sep 2014 10:32:55 GMT

Have your SAS session run a utf-8 one. You will better use several connections (possible with server side) with utf8/latin-1 encodings.

Using a utf-8 sas session will require attention to character functions as not all of them are valid with utf8.

SAS(R) 9.4 National Language Support (NLS): Reference Guide, Third Edition

Would go for 9.4 with SAS/connect better supporting the interconnections.

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Thu, 11 Sep 2014 02:49:58 GMT

I appreciate your help! I realized the mistaken was something wrong with encoding.But how can I have my SAS session run a utf-8 one ? The encoding of my SAS server side is latin1 default.How can I change the configuration about encoding？

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Patrick — Thu, 11 Sep 2014 03:00:47 GMT

Below two links should provide the information you're after.

SAS(R) 9.3 National Language Support (NLS): Reference Guide

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Thu, 11 Sep 2014 03:01:56 GMT

I found a solution on internet.The solution says adding the latin1 encoding to sasv9_usermods_cfg file.But I don't know how to do that.

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Patrick — Thu, 11 Sep 2014 03:13:50 GMT

You need to have admin privileges to change the .cfg. But have a look at the links I've posted. Some of the options work also as part of an options statement.

This bit in the docu for an option tells you where you can set/change it. If it says "configuration file, SAS invocation" then it's the .cfg or the autoexec, if it's "OPTIONS statement" then you can also set it in your code.

Valid in:

configuration file, SAS invocation, OPTIONS statement, SAS System Options window

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Thu, 11 Sep 2014 06:04:24 GMT

I reset the server side and deployed it as a unicode server.The error was solved.Maybe your answer also can solve this error,But the time is not enough for me.Thank you so much!

I also want to ask one question.

Now,I have a client,a sas application server and a database server(DB2),They are located in different computers.

I can use 'libname' statement to connect to the database.Can I directly analyse the data on the logic table from database?(the error :failed to I/O )

Should I move the data to the sas application server and then to analyse?

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jakarman — Thu, 11 Sep 2014 07:18:53 GMT

Multilanguage is becoming better. http://support.sas.com/resources/papers/Multilingual_Computing_with_SAS_94.pdf

Still you have to deal with an en (English latin1 single byte) and u8 (utf8 multibyte) sas session. To avoid compatiblity issue uses several appservers

http://support.sas.com/resources/papers/AddingAdditionalSASWorkspaceServerstoSupportMultipleEncodings.pdf

Having done a serverside installation supporting Unicode and en language it is small configuration step to add an addtional server and change the sas-scriptname that will be used for the app-server (u8 or en) in the usermods-files.

Before 9.4 connections of sessions with different encoding is a disaster SAS/CONNECT(R) 9.2 User's Guide (Connect) and Proc cport / cimport also fails.

As you are mentioning DB2, that one is often limited to latin-1. When that is the case the DB2-client is ok.

Going for DBMS supporting Unicode is posing an other issue. The RDBMS client will possible give back 4 bytes when in utf8 mode. With latin1 you need just one. As the DBMS client does this translation you possible need two configurations of the DBMS client.
Check this when you are getting into unexpected growth of char variables. (multiplied by 4).

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jakarman — Thu, 11 Sep 2014 08:08:43 GMT

Needed to process your DB2 question. When you are connecting that by a libname (db2 client configured and sas/access DB2) working, you can use that as input for analyses. just take care for the datatypes: SAS/ACCESS(R) 9.4 for Relational Databases: Reference, Fifth Edition

Wanting to do more with that interface you could use DB2 specific functions with explicit pass through or going for DS2.
DS2 is new world for accessing modifying data. SAS(R) 9.4 DS2 Language Reference, Third Edition (if you have time left to spend)

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Thu, 11 Sep 2014 08:39:44 GMT

I have successfully connected the database using 'libname' statement like following picture(can see the tables):

then there are two errors when I use this code;

this is my database server code set(gbk):

So the reason why cause errors is the encoding of database is gbk and the application server is unicode？

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jakarman — Thu, 11 Sep 2014 09:09:18 GMT

You have a working connection nice.
It wil get the data to the sas-dataset work.test , then putting data there coming from db2 is running out of space of the saswork
SASwork is a standard name in SAS for temporary data. There is an other utilloc that is special for utilities sort sql.

You analyses will probably run with small data (eg obs100).

I do not know how you configured that work, but it is running out of space.

Can you check the physical name and sizing on the server of that one? To get the information on that:

libname work list ; /* shows all attributes */

proc options; run; /* shows all std options look for work */

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

Patrick — Thu, 11 Sep 2014 10:07:13 GMT

As points out you're not having enough disk space in your SAS work area.

To answer one of your other questions: Yes, it's a good idea to push processing to the data base. The normal approach is to first reduce volumes on the data base and then only move the data you actually need over the network into SAS.

For your code: If you can, select only the columns you need (so no Select *) and subset the rows if possible (where ....). Also should you only need summary values then push this bit as well to the data base (eg. using a group by statement in your SQL code).

Some SAS procedures allow you to push all processing to the data base (eg. Proc Freq and Proc Means). So if you can get your desired result with these procedures then use them.

Using below options shows you in the log where processing happens SAS/ACCESS(R) 9.4 for Relational Databases: Reference, Fifth Edition

options sastrace=',,,d' sastraceloc=saslog nostsuffix;

There are also other techniques for improving performance when dealing with data stored in a data base. If you're working regularly with DB2 then it might be worth to spend some time and get up-to-speed with these techniques.

Here only one of many possible links: http://support.sas.com/documentation/cdl/en/acreldb/67473/HTML/default/viewer.htm#n13m5rcjajo3c5n12lcdabpyaei9.htm

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Fri, 12 Sep 2014 02:22:43 GMT

yes.I checked the information about my SASWork.

it's too small to copy the whole data from database.How to enlarge the size of SASWork?

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Fri, 12 Sep 2014 02:36:21 GMT

Thanks a lot.I learn much about SAS this time.

I hope you and can help me when I face new problems.:smileylaugh:

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jakarman — Fri, 12 Sep 2014 06:58:47 GMT

gingyu, It is amazing how easy and far you are doing with this not easy SAS challenges. Not being mindblocked, solving each topic one by one in multitasking mode.

That /tmp location is the default for a Unix temporary files (Posix Linux apology as I use the old AT name) . I do not kow how many users you have on the system and what your options are for file-systems and storage. You could extend the /tmp filesystem but as it is the common for all tmp files on the machines may not the best way. You could set up some new mountpoints like:

/var/local/sas/tmpusr -> for personal users the workspace server appserver

/var/local/sas/tmpbtch -> for shared operations batch processing of an it-operations like

SAS(R) 9.4 System Options: Reference, Third Edition (work)

/var/local/sas/tmputl -> for the interenal tmporary files of Proc sql, proc sort. These filesnames are named #UTL.... normally invisible

SAS(R) 9.4 System Options: Reference, Third Edition (utillloc)
You do not possible need all of them it is only for loadbalancing having multiple users/processes running on the system

Every mountpoint/filesystem can be setup with enough space (Unix administration) for your intended processing. Any idea of your biggest datset sizing you will proces? make it at least 4 times bigger than that. (I forgot) set the acces rights on those directory like /tmp. A chmod of 3777 or 3770 will do setting the sticky bit and groupid bit. If a group access can be used to limit others can left out.

Check those filesystems when running out of space. Regular Cleanup of workfiles can be automated (using cron) later, something to park in your mind now.

Having those filesystems in your unix you can point to them using the sas-config-files. (Lev-/SASApp\sasv9_usermods.cfg). dedicated nested one can be used to give a SP or batch-server an other work-location

-work /var/local/sas/tmpusr

-utilloc /var/local/sas/tmputl

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Fri, 12 Sep 2014 08:53:40 GMT

I appreciate your patient.I benefit greatly from you.

Thanks a lot!

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Mon, 15 Sep 2014 07:20:49 GMT

It's me again.I have a new problem.

I want to test the highest pressure of my database server and SAS server(the maximum of SAS users) .

I have no idea about that.

Could you give me some advice or methods?

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jakarman — Mon, 15 Sep 2014 09:26:19 GMT

Gingyu, This question is a more difficult one not for the technical questions, but by its concepts wit all involved uncertainties.

The maximum of Users is very dependent on a lot of factors. The used Hardware, type of SAS usage, optimizations performance/tuning.

Testing of your implementation how it is performing can be seen as something as kind of regression testing. When planning for a kind of service you should set some kind of benchmark dedicated for your type of service.

Hardware:
You started with an installation and are asking now what it could serve. It is an approach workable in a new environment. In an existing environment you are knowing some of the usage and with that an sizing and than start with the sizing and tuning of the hardware (memory cpu io) with the operating system (OS). Clustering and Grid to be evaluated as options.

SAS usage:

There are several types of usage:

- I interactive The classic information consumers running some known "sas application"

- B batch The classic data processing as executed by an operations support staff. t strategy (Agile waterfall)

- D descriptive adhoc analytics It is the BI part for the next question to answer or self-service BI

- M data/etxt mining

note:

I,B will use a Develop/Test/Acceptance/Production (DTAP) release managemen

D,M will not use the classic DTAP as it developed with production data. There will be a need to segregate
I,D are by nature executed in office hours.

B,M can be scheduled less or more to run outside office hours.

Every usage type can be divided to some ordinals of resources usage:

- light

- medium

- heavy

- extra ordinary

It is about memory/cpu/io and expected runs active at the same moment with a response-time and the DBMS load.

With interactive light-usage there will be a small probability of parallel running. By this you can support a lot of users.

Extra ordinary data mining on the other can be that very demanding that running a couple of uses can exhaust the same machine.

It is good to know about the existence of cgroups - Wikipedia, the free encyclopedia. It is kernel change where it is made possible to do load balancing. It is rather new and not yet accepted as common practice in the Linux world.

By this you have a 4*4 matrix of resource usage with some expectations with assumptions to be filled in.
Knowing the load capacity of your machine/configuration you can set some predictions on the expected load with users.
It will be a continuous measuring and refining to optimize this model. Changing a badly build "sas application" can move that one from heavy to light. To do such things is balancing cost/effort. As you see there will be no fixed defined way how to work at this load cost change performance area.

Resource usage:
Knowing your environment it should be easy to design some tests that will easily overload you machine.

Using Eguide (starting with 5.1) or SAS/connect (MP connect) you can start by yourself a lot of task's in parallel.

Running the same or slightly change code you can get some impression on the performance.

Simple tests:
- Running a proc sort on some faked generated data will cause a heavy load on IO and/or memory using cpu.

having the sizing parameterized you can do this for 1G 10G 20G 50Gb 100Gb sizes dataset. And do this for 1-10 jobs in parallel.

For sure the machine should get overloaded at some moment.

- For the DBMS you can imagine some of the same making difference between reading the DBMS and writing the DBMS

- (more)

You probably did not tuned your SAS options. Sumsize (proc means) memsize bufsize aligniofiles. Some of them will have impact on the bigger data processing.

In a multiuser system it is nor very sensible to give all memory to one process. You can run into http://en.wikipedia.org/wiki/Thrashing_(computer_science).

Connecting to your dbms think on: defer=yes bulkload bufsize. The OS (tcpip) is tuned for 4k block others often for the 64K block sizes.

There is some choice to make, I am used to go for data processing for bigger data, accepting causing some overhead for the smaller ones.

As example a faked dataset, easy to modify to your own ideas. This faked dataset in uncompressed setting is about 1Kb/record. Using the compress=binary is a good tip for datasets having more data fields. If would suggest using an array of 256 character variables 8 bytes long with some random content.

data ivp_run.tst1 ;

length tekst $1024;

do i=1 to &sizenum ;

var1=I; var2=ranuni(10000) ; tekst="äsb 1234567890 qwerty" ; output;

end;

run;

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jackhost — Fri, 19 Sep 2014 03:13:38 GMT

Hi it's me again.

As your answer about changing the SASWork.I faced a new error;

I have defined follows:

/CUBEDATA/LVCUBDAT1/SAS/SASHome/tmpusr

/CUBEDATA/LVCUBDAT1/SAS/SASHome/tmputl

tmpusr file contains:

/CUBEDATA/LVCUBDAT1/SAS/SASHome/sastempfiles

method=space

tmputl file contains:

/CUBEDATA/LVCUBDAT1/SAS/SASHome/sastempfiles

method=space

Have written into Lev-/SASApp\sasv9_usermods.cfg

-work /CUBEDATA/LVCUBDAT1/SAS/SASHome/tmpusr

-utilloc /CUBEDATA/LVCUBDAT1/SAS/SASHome/tmputl

and restart the SAS servers.

But the error is like following pictures:

Re: Failed to transcode data from U_UTF8_CE to U_LATIN1_CE

jakarman — Fri, 19 Sep 2014 05:39:16 GMT

You have defined:

/CUBEDATA/LVCUBDAT1/SAS/SASHome/tmpusr

/CUBEDATA/LVCUBDAT1/SAS/SASHome/tmputl

These should be visible and accessible using the key/account for your workspace server.

The modifications are only for SASApp that is the WS/SP services (running by sasssrv , personal account ).

The Object spawner and metadataserver are untouched not needing any restart.

You are getting an error that is telling you something is wrong having done that change. Now we need to find what is wrong.
1/ switch to that personal account and verify

+ you can go into that directory every directory in de hierarchy should at least have x access (group/others) seeing the content needing r-access(group/others).

There are a lot of people "security specialist" not knowing the x right is indicating you are able to do the "cd" command. That is needed for the whole hierarchy:

= /CUBEDATA

= /CUBEDATA/LVCUBDAT1

= /CUBEDATA/LVCUBDAT1/SAS

= /CUBEDATA/LVCUBDAT1/SAS/SASHome

= /CUBEDATA/LVCUBDAT1/SAS/SASHome/tmpwrk

+ create new files directories there (tmpwrk).

The chmod rights (do a ls -la) should be 3770 of 3777 (rwxrws--T rwxrwsrwt)

Those letters s and t are no typos, they are comin from the Gid-bit setting (that 2 first of 4 digits) and sticky-bit settings (that 1 giving 3 first of 4 digits).

That gid-bit will cause the group inheritance from the directory, not the first group associated with the active key

That sticky bit will cause onely the owner of the file or the owner of the directory are able to delete files. This is by default set on "/tmp"

There are a lot of people "security specialist" not knowing this.

As you can create files there with a restricted personal key, than you have validate this.

2/ double check the spelling of all the directory names. Unix is case-sensitive. Every spelling with a caps/nocaps difference is a real difference.

As this is different in Windows and in SAS (as long it is SAS language specific) this can be a behavioral pitfall.

IT could be a typo somewhere, check the name using the output of "pwd" command with the SAS-config modification in the SASApp

3/ You could use some terminal access using sas to verify basic settings. It is only doable when you are good in sas knowing all of that basic modes.

he sas Linux-script can be manually started,

- it will try the dms interface using x11. You will need a x11-server for that.

- with the options -nodms -noterminal it will start a line-mode interface showing a number where you can put a sas-statement. "endsas; " is the statement toe leave that.

You can leave you Eguide open and there is no need for restarting the objectspawner/metdataserver.

As long you have the error there will be no connection. Watch the processes being started by you when you get the connection