topic Re: Handling emojis in JSON in SAS Programming

Handling emojis in JSON

sindrew — Mon, 14 Sep 2020 08:16:21 GMT

I'm trying to convert json from a rest api to a dataset. In the JSON data there is an emoji, which SAS, or the proc json routine is unable to handle. I've played with idea of getting the data with python (don't know why), but not sure if python will handle this better.

Example code:

filename input   "testjson.json";
%let jsondata = '{"num_records": 1, "data": [[1, "texttexttext \ud83d\udc4d\ud83c\udffb"]], "data_type": "json"}';

libname posts json fileref=input;
proc datasets lib=posts ; quit;

The data array includes more attributes, but libname statement fails on the 4 encoded values at the end of the provided example and program throws the following error. The field is provided by users so there may be more...

ERROR: Invalid JSON in input near line 1 column 71: Some code points did not transcode.
ERROR: Error in the LIBNAME statement.

Is it possible to handle this in some way besides getting the developer to change their API (probably not feasible)?

Re: Handling emojis in JSON

yabwon — Wed, 23 Jun 2021 16:31:01 GMT

hi,

is the `jsondata` the content of the file?

I executed the following code on sas 9.4m6 (running with ENCODING=UTF8), to put the data into the text file:

filename input "testjson.json";

data _null_;
file input;
put '{"num_records": 1, "data": [[1, "texttexttext \ud83d\udc4d\ud83c\udffb texttext"]], "data_type": "json"}';
run;

libname posts json fileref=input;
proc datasets lib=posts ; quit;

proc print data = posts.alldata;
run;

proc print data = posts.data;
run;

proc print data = posts.root;
run;

and everything seems to be ok, log was clear and in the proc print I saw 👍 icon (thumbs up)

All the best

Bart

Re: Handling emojis in JSON

sindrew — Mon, 14 Sep 2020 09:12:22 GMT

Hi bart

Thanks, 👍 to 👍. Yes jsondata is the content of the file (a bit unclear).

I'm on 9.4m5. Possible fix ther....but could encoding be an issue? I'm using wlatin1.

Re: Handling emojis in JSON

yabwon — Mon, 14 Sep 2020 09:15:31 GMT

My first approach would be to run UTF-8 sSAS session for utf-8 data.

Bart