BookmarkSubscribeRSS Feed
0 Likes

Along the lines of having better control of messages in the log, provide a way to disable warnings when a Hadoop string is 32k in length.

If I run:

proc sql;
  connect to hadoop ( &hadoop_connection.);
  select * from connection to hadoop ( describe extended TABLE );

I get the message:

WARNING: The following columns could have a length in SAS of 32767. If so, SAS performance is impacted. See SAS/ACCESS documentation for details.  The columns read from Hive followed by the maximum length observed were:  col_name:450, data_type:639, comment:0

 

I should be able to query to Hadoop metadata without warnings.

 

 

4 Comments
Tom
Super User
Super User

Does using either the DBMAX_TEXT or DBSASTYPE settings help?

ChrisNZ
Tourmaline | Level 20

Sadly no.

I am not using a library, so can't use DBMAX_TEXT.

I am not reading a table, so can't use BMAX_TEXT or DBSASTYPE.

Also, since there is no table, extended table attributes (SASFMT) are out as well.

ChrisNZ
Tourmaline | Level 20

Mmm I tried another syntax and it works .. sort of.

libname HAD001 hadoop &hadoop_connection. DBMAX_TEXT=1024;

proc sql;
  connect using HAD001;
  select * from connection to HAD001 ( describe extended TABLE);   

Does not create a message.

ChrisNZ
Tourmaline | Level 20

I suppose a workaround is

1- Use the "connect using" syntax instead of the "connect to" syntax.

2- Set DB-MAX_TEXT=32766 to fool the warning generator.

libname HAD001 hadoop &hadoop_connection. DBMAX_TEXT=32766;

proc sql;
  connect using HAD001;
  select * from connection to HAD001 ( describe extended TABLE );   

But really, a better way would be desirable.

Thanks a lot @Tom for getting me out of the rut!