Data insertion issue for other languages from CI communication definition to CDM

[nikhil_mahajan]

Hello All,

I am collecting user inputs through campaign definition in other than English languages, CI accepts the inputs but on execution/publish o campaign data is not properly inserted into Common data model. It got inserted in the form of special non readable characters.

Please suggest.

Regards,

Nikhil

[Dmitry_Alergant]

Hi, it's a fairly typical situation caused by a wrong choice of encoding of SAS 9.4 platform environment. Indeed, everything that happens inside the app (CI Studio) is not affected as it's all bor built in Java and natively based UTF-8, but the data processing through SAS Platform (Compute Server) that includes writing to CDM corrupts the data when encoding is not chosen correctly there.

Depending on what languages you need to support, you'll need to enable either an appropriate single-byte encoding (Western European, Cyrillic, etc.) - or UTF-8 to support all languages at once.

There are many places where encoding can be specified that affect different layers and areas of the platform, arguably best is the top-level config file in SASHome/SAS9.4/SASFoundation/sasv9_local.cfg   (option encoding = ...)

Switching between single-byte encodings is relatively straightforward. Switching from a single-byte to UTF-8 is a more impactful process and will require several additional steps, including explicit conversion of some important datasets. You can try coordinating this with the help of SAS Technical Support, or a consulting partner you are working with. 

Also, switching from a single-byte to UTF-8 encoding may lead to some degree of performance degradation due to significantly increased workload on WORK storage, and some compatibility issues when users are creating calculated items with string functions. Some calculated items will need to be updated in order to work correctly. In brief, if your language requirements situation allows you to stay within a single-byte encoding rather than switching to UTF-8, it's often much simpler.