Is there a routine to converte Hebrew text to Unicode in an Annotate data set?

[Shmuel]

@JonathanNitzan I edited my last code. It supplies macro variables in the form you need:

data test;
  length strin: $80;
  infile datalines truncover dlm='09'x; /* use TAB as delimter betweem varname and string */
  input varname $ string_in :$char80.;
  string_out = compress(unicodec(string_in), '\u');
  call symput(varname,strip(string_out));
  keep varname string_out;
datalines;     /*** use TAB between VARNAME and the string to translate ***/
ahuzim	אחוזים
std1	+ 2 סטיות תקן
std2	+ סטית תקן
std3	- סטית תקן
std4	- 2 סטיות תקן
avg		ממוצע
mikra	מקרא
month	חודש
total	סה"כ
;
run;
%put Ahuzim=&ahuzim; 

[JonathanNitzan]

Shmuel: This is the log I get from running your code (after retyping אחוזים and inserting a tab as needed). There must be something amiss in my system or in what I'm doing...

2101  data test;
2102    length strin: $80;
2103    infile datalines truncover dlm='09'x; /* use TAB as delimter betweem varname and string
2103! */
2104    input varname $ string_in :$char80.;
2105    string_out = compress(unicodec(string_in), '\u');
2106    call symput(varname,strip(string_out));
2107    keep varname string_out;
2108  datalines;

NOTE: The data set WORK.TEST has 9 observations and 2 variables.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.00 seconds


2108!                /*** use TAB between VARNAME and the string to translate ***/
2118  ;
2119  run;
2120  %put Ahuzim=&ahuzim;
Ahuzim=

[Shmuel]

It is difficult to guess why you got "Ahuzim=    " ? 

Please post the code as submitted. Next is my log - using SAS UE:

1          OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
 72         
 73         data test;
 74           length strin: $80;
 75           infile datalines truncover dlm='09'x; /* use TAB as delimter betweem varname and string */
 76           input varname $ string_in :$char80.;
 77           string_out = compress(unicodec(string_in), '\u');
 78           call symput(varname,strip(string_out));
 79           keep varname string_out;
 80         datalines;
 
 NOTE: The data set WORK.TEST has 9 observations and 2 variables.
 NOTE: DATA statement used (Total process time):
       real time           0.03 seconds
       cpu time            0.02 seconds
       
 
 80       !                /*** use TAB between VARNAME and the string to translate ***/
 90         ;
 91         run;
 92         %put Ahuzim = &ahuzim;
 Ahuzim = 05D005D705D505D605D905DD
 93         %put STD1 = &std1;
 STD1 = + 2 05E105D805D905D505EA 05EA05E705DF
 94         %put MIKRA = &mikra;
 MIKRA = 05DE05E705E805D0
 95         
 96         
 97         OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
 109        

[Shmuel]

You can brows also the created dataset TEST and check does string_out contains the expected data

[Shmuel]

What sas platform are you using? and what sas version do you run?

[JonathanNitzan]

Regarding your questions, Shmuel:

1. SAS 9.4 TS Level 1M3, X64_8Home

2. I checked the test data set, and the variable string_out is empty for all values.

Jonathan

[Shmuel]

Have you tried my previous method, code starting with:

/* tarnslate hebrew text to UNICODE */
%let kbd_aleph = א;    /* RETYPE with local keyboard */
%let uni_aleph = 1488; /* ALEF uncode = '05D0'x */
%let ch_len = 2;

data Heb_Table;
  length string_in string_out $130     /* Hebrew 2 bytes/char include spaces */
         aleph $2;
  retain aleph "&kbd_aleph" delta;
  if _N_=1 then do;
     delta = rank(substr(aleph,1,1))*256 + rank(substr(aleph,2,1)) - &uni_aleph;
     put DELTA=;
  end;
...........

what output you get?

[Shmuel]

@JonathanNitzan ,

1) Pay attention to Eyal's answer.
    I'm using SAS UE - proc options result in next log:

73         proc options option=encoding;run;
 
     SAS (r) Proprietary Software Release 9.4  TS1M6
 
  ENCODING=UTF-8    Specifies the default character-set encoding for the SAS session.

2) Have you tried my first provided method ?

    Do you have issues with it?

3) Next is a third method to create macro variables.

    It consists of 2 parts:

     -1- Creating a FORMAT $h2u used for translation

data heb2uni; 
  retain fmtname '$h2u';
  length heb $2 uni $4;
  infile cards;
  input uni $ heb $;
cards;
05D0 א
05D1 ב
05D2 ג
05D3 ד
05D4 ה
05D5 ו
05D6 ז
05D7 ח
05D8 ט
05D9 י
05DA ך
05DB כ
05DC ל
05DD ם
05DE מ
05DF ן
05E0 נ
05E1 ס
05E2 ע
05E3 ף
05E4 פ
05E5 ץ
05E6 צ
05E7 ק
05E8 ר
05E9 ש
05EA ת
; run;
proc format lib=WORK    /* better choose LIBRARY instead WORK */
     cntlin=heb2uni(rename=(heb=start uni=label));
run;

     If you changed to lib=LIBRARY you can run that part once.

     The format will be kept for next sessions too (Depending on your

     SAS platform - not available in SAS UE).

     -2- Translating Hebrew strings and assign the result to macro variables

           you named, same as in previous two methods:

data HebTable;
   length hebstr  unistr $80 ch $4;
   infile cards dlm='09'x truncover;
   input varname $ hebstr $;
   unistr = ''; hebstr = strip(hebstr);
   do i=1 to length(hebstr) by 2;
      ch = put(substr(hebstr,i,2), $h2u.);
      unistr = cats(unistr,ch);
   end;
   *putlog varname=  hebstr= unistr=;   /* unmark for test */
   call symput(varname,strip(unistr));
cards;   
ahuzim	אחוזים
std1	+ 2 סטיות תקן
std2	+ סטית תקן
std3	- סטית תקן
std4	- 2 סטיות תקן
avg		ממוצע
mikra	מקרא
month	חודש
total	סה""כ
;
run;

Pay attention - if Hebrew string contains a quote (') or double quotes (")

type it twice as shown in TOTAL סה""כ - to avoid syntax error.

Check results, either by:

%put AHUZIM = &ahuzim;
%put STD1 = &std1;
%put AVG = &avg;

LOG shows:

 133        run;
 134        %put AHUZIM = &ahuzim;
 AHUZIM = 05D005D705D505D605D905DD
 135        %put STD1 = &std1;
 STD1 = +205E105D805D905D505EA
 136        %put AVG = &avg;
 AVG = 05DE05DE05D505E605E2

OR by simulation to what you need/use:

data _null_;
  txt1 = "(xAHUZIM= &ahuzim xxx)";
  put txt1=;
  txt2 = "(xTOTAL= &total xxx");
  put txt2=;
run;

LOG shows:

 txt1=(xAHUZIM= 05D005D705D505D605D905DD xxx)
 txt2=(xTOTAL= 05E105D4"05DB xxx)

Pay attention, though the double quotes were typed twice it appears once only.

[JonathanNitzan]

Thank you Eyal and Shmuel.

1. I think I've managed to lose track of all the different solutions, at least in part because my system does not recognize the Hebrew text I correctly typed in.

2. Following your recommendation, I ran the following code:

proc options option=encoding;run;

and got the following Log:

95   proc options option=encoding;run;
    SAS (r) Proprietary Software Release 9.4  TS1M3
 ENCODING=WLATIN1  Specifies the default character-set encoding for the SAS session.

So I get encoding=WLATIN1 instead of the desired encoding=Whebrew. I suspect this difference explains the ??? replacing the typed-in Hebrew text.

3. What should I do the change my encoding?

Thank you.

[Shmuel]

Option encoding is a sas system option and have to be defined at sas startup.

You can change it either in the sas configuration file (for online and for batch executing) or

at exec sas (batch only).  Try first encoding="UTF-8"; 

Which OS do you use? Is it Windows?

I believe Eyal can guide you better the me. 

The easiest way to check if the change works fine is by running

data _null_;
  text = 'אבגדה';
  put text=;
run;

you should see in the log the same Hebrew text as typed (and not text=?????).

[JonathanNitzan]

1. I've followed the instructions to change my SAS encoding  as indicated here:

https://communities.sas.com/t5/SAS-Communities-Library/Change-Encoding-using-SAS-Enterprise-Guide-Wi...

2. I've run proc options option=encoding;run; I still get WLATIN1

3. I'm using Windows 10.

Jonathan

[Shmuel]

1) Are you using SAS Enterprise Guide?

2) Please post the configuration file after editing, changing to encoding=UTF-8

3) Did you closed sas before editing the .cfg file ? then run a new sas session and check.?

[JonathanNitzan]

1. I'm running a SAS session. I don't think I'm running SAS Enterprise Guide. This is my Log after starting SAS:

NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software 9.4 (TS1M3)
      Licensed to YORK UNIVERSITY, Site 70084392.
NOTE: This session is executing on the X64_8HOME  platform.
NOTE: Updated analytical products:

      SAS/STAT 14.1
      SAS/ETS 14.1
      SAS/OR 14.1
      SAS/IML 14.1
      SAS/QC 14.1

NOTE: Additional host information:
X64_8HOME WIN 6.2.9200  Workstation

NOTE: SAS initialization used:
      real time           1.82 seconds
      cpu time            1.51 seconds

2. This is the configuration file after editing:

-config "C:\Program Files\SASHome\SASFoundation\9.4\nls\u8\sasv9.cfg"

3. This configuration file was created when SAS was closed. After restarting SAS, I got:

1         proc options option=encoding; run;

    SAS (r) Proprietary Software Release 9.4  TS1M3

 ENCODING=WLATIN1  Specifies the default character-set encoding for the SAS session.

[Shmuel]

1) I mentioned SAS EG because the link you used is:

     "https://communities.sas.com ... /Change-Encoding-using-SAS-Enterprise-Guide-Windowing-Environment/ ... ".

 2) Is sasv9.cfg the only configuration file in that folder?

     Getting ENCODING=WLATIN1 after updating the configuration file means -

         * either you forgot to save the file after updating 

         * or WLATIN1 maybe is the default value in case encoding was assigned with a wrong code.

           Check valid encoding values in the documentation.

         * or encoding appears again somewhere else, maybe in the autoexec.sas file

            if there is such (I'm not sure it is valid there). 

         * or it appears while invoking sas.

     That is the reason I asked to post the .cfg file, and I meant the contents not just the path

     and name in order to check it.

     SAS System options may appear in each of next sites, and the last appearance is used

        - the cfg file

        - at invoking sas - exec sas <options>

        - the autoexec file, if it is valid there

I'm using now SAS UE and I don't have accessibility to the configuration file which is on a remote server somewhere in the cloud.

[GraphGuy]

Looks like you guys have got it figured out (use utf8 SAS), but here's some extra info that might be useful...

You might be interested in the $uesc200. format...

In the mapsgfk.*_attr attribute datasets for the maps, we sometimes include a \u version of the text, to support the special characters in the local versions of the text/names. For example, see the idnameu variable in the mapsgfk.israel_attr dataset. (the reason we store them like this is so that people can use these same datasets in both utf8 and non-utf8 SAS sessions).

proc print data=mapsgfk.israel_attr;
var idnameu;
run;

If you're running utf8 SAS, then you can use the $uesc200. format to convert the \u characters to the actual local characters you will recognize.

data foo (keep = idnameu newtext_utf8); set mapsgfk.israel_attr;
newtext_utf8=input(idnameu,$uesc200.);
run;
proc print data=foo;
run;

And, if you have local characters, and you want to convert them to \u characters (so you could work with them in a non-utf8 SAS session), you can use the format to go the other way too ...

data foo; set foo;
newtext2=put(newtext_utf8,$uesc200.);
run;
proc print data=foo;
run;