topic Re: Hidden blanks in SAS Programming

Hidden blanks

dcortell — Tue, 21 Jun 2022 15:21:45 GMT

Hi all

I have a dataset which is the results of

web scraping from html pages text and
then split each text into bigrams (set of two characters)

So each line is a combination of a term (example: "ba", "ca","de" etc...) and the length of the term (which is max 2 and min 0 if we are extracting the last characters from a word)

Example dataset

Now, in order to avoid to have spaces in the bigrams, but to still keep track of them, I'm substituting them for underscores

proc sql;
create table test01 as
select *, length(_term_) as length_bigr,
length(_term_2) as length_bigr2
from move.language_det_NGRAMS
where iden=1;
quit;

The problem is, I'm getting weird results as output, as you can see in the image above

The _term_2 is filled with underscores till reaching the max length for a text var (200)

I'm not sure what is causing this. Could be some html hidden format/character? Someone can share knowledge about if some pre-processing is still to be made on the text derived from the html to get rid of hidden spaces or something related?

Bests

Re: Hidden blanks

PaigeMiller — Tue, 21 Jun 2022 15:33:02 GMT

How is _TERM_2 created in data set move.language_det_NGRAMS? You don't show us the code where it is created.

Re: Hidden blanks

Kurt_Bremser — Tue, 21 Jun 2022 15:32:54 GMT

Somewhere along the way, _term_2 was defined with a length of 200. SAS character variables are always padded with blanks up to their defined length (if they only contain blanks, it is a missing value per definition).

Re: Hidden blanks

dcortell — Tue, 21 Jun 2022 15:55:08 GMT

Sorry, _term_2 is created as follows:


data move.language_det_NGRAMS;
set move.language_det_NGRAMS (drop=_term_2);
_term_2=tranwrd(compress(_term_,,'kw')," ","_");
run;

Re: Hidden blanks

Tom — Tue, 21 Jun 2022 16:14:44 GMT

So you never set a length for _TERM_2. Instead you forced SAS to GUESS how you wanted it defined.

It should define it to match how _TERM_ is defined. So if you are seeing _TERM_2 be defined as length $200 then _TERM_ was probably defined as length $200 also.

What is the purpose of the COMPRESS() function call? If you want to just remove leading/trailing spaces you could use STRIP().

Why use TRANWRD() when just converting single characters. You can use TRANSLATE() for that.

What if you use something like this instead:

data move.language_det_NGRAMS;
  set move.language_det_NGRAMS (drop=_term_2);
  length _term_2 $2 ;
  _term_2=translate(strip(_term_),'_',' ');
run;

Re: Hidden blanks

dcortell — Tue, 21 Jun 2022 16:26:30 GMT

So this is how _term_ is generated from the text:

data move.language_det_NGRAMS (where=(_i_=2));
   set viyadrop.language_detect_train2;
   _tmpStr_ = cleaned;

   do while (klength(_tmpStr_)>0); 
   /**max is min betweeb length or 2: size of grams****/ 
      _maxN_=min(klength(_tmpStr_), 2); 
    
      do _i_=_maxN_ to _maxN_;
    /**extract from pos1 n of words _i_ (3): 1 to 3 trigram **/
         _term_ = ksubstr(_tmpStr_, 1, _i_);
         output;  
      end;  
   
  /** after extracting trigrams from pos 1,
  if length >1 then temp=move to next pos **/
   if klength(_tmpStr_)>1 then _tmpStr_ = ksubstr(_tmpStr_, 2);  
      else _tmpStr_ = '';
   end;
  
   keep iden class _term_ _i_;
run;

Re: Hidden blanks

dcortell — Tue, 21 Jun 2022 16:29:44 GMT

I can't use strip as, example of text "A bird". The set of bigrams should be: "A_" - "_b" - "bi" - "ir" -"rd". If I employ strip, the "A_" bigram will come up as "A" as the blank will be removed in the strip phase

Re: Hidden blanks

dcortell — Tue, 21 Jun 2022 16:36:47 GMT

Also, I have tried the following:

data test;
set move.language_det_NGRAMS  ( obs=20 drop=_term_2);
_term_2=translate(compbl(_term_),'_',' ');
run;

proc sql;
create table test01 as
select *, 
length(_term_) as length_bigr,
length(_term_2) as length_bigr2
from test
where iden=1;
quit;

But for same reason the result is not the expected: Example

In bigrams as the one in line 4, where there is only a combination of "blank" and "h", I would expect the above should output "_h", but then instead an underscore is also filled on the right side of the character, despite the length being only 2 of _term_

Re: Hidden blanks

PaigeMiller — Tue, 21 Jun 2022 16:43:58 GMT

@dcortell wrote:

I can't use strip as, example of text "A bird". The set of bigrams should be: "A_" - "_b" - "bi" - "ir" -"rd". If I employ strip, the "A_" bigram will come up as "A" as the blank will be removed in the strip phase

Then use TRIM()

Re: Hidden blanks

Tom — Tue, 21 Jun 2022 16:53:06 GMT

So you want to split the string into strings of exactly two characters (not two bytes)? You want to split to be overlapping? So a string like 'ABCD' becomes 'AB', 'BC', 'CD' and 'C ' ?

Use KSUBSTR() to limit the sub term to only two characters. Make sure to use KTRANSLATE() since you are dealing with potential multibyte characters.

data want;
  set have;
/* A unicode character can use up to 4 bytes */
  length term $8 ; 
  do index=1 to klength(string);
    term = ktranslate(ksubstr(string||' ',index,2),'_',' ');
    output;
  end;
run;

Re: Hidden blanks

dcortell — Tue, 21 Jun 2022 16:50:23 GMT

Sorry but it is the same, if I use Trim() or Strip(), the trailing blanks will be removed, and the bigram "A " which should transform in "A_", would just end up with the trim intermediate step to output as "A" as the leading space will be removed. The problem is not in removing trailing or leading blanks in the bigram, but understand why they seems get "multiplied" if we use then Translate(_term_,"_","") function

Re: Hidden blanks

Kurt_Bremser — Tue, 21 Jun 2022 16:52:39 GMT

From the documentation of the TRANWRD Function:

Length of Returned Variable

In a DATA step, if the TRANWRD function returns a value to a variable that has not previously been assigned a length, that variable is given a length of 200 bytes. You can use the LENGTH statement, before calling TRANWRD, to change the length of the value.

Re: Hidden blanks

dcortell — Tue, 21 Jun 2022 17:05:38 GMT

@Kurt_Bremser , ok, but also switching as suggested to the use of translate() in combination with compbl() as per the example above, does not explain why still blanks are added where it should be not the case. As per documentation:

"In a DATA step, if the TRANSLATE function returns a value to a variable that has not previously been assigned a length, then that variable is given the length of the first argument."So also trying this direction, and looking at the example mentioned of line 4, where we have a " h" bigram, the above translate(compbl(),"_"," ") should output "_h", but then we get a "_h_" output, adding another spaces at the left of the character

Re: Hidden blanks

ballardw — Tue, 21 Jun 2022 17:22:36 GMT

Perhaps you will find this example a bit enlightening as to what is happening:

data junk;
   length _term_ $ 10.;
   _term_='ha';
   x = compbl(_term_);
   _term_1=translate(x,'_',' ');
   _term_2=translate(strip(x),'_',' ');
   _term_3=translate(strip(compbl(_term_)),'_',' ');
run;

_term_1 is basically what you are doing.

Re: Hidden blanks

dcortell — Tue, 21 Jun 2022 17:35:24 GMT

Terrific. This perfectly solve the topic. Thanks!