when using libname jdbc to connect to hive an error appears

duanzongran — Wed, 10 Jan 2024 09:37:17 GMT

dear all:

I use jdbc to connect to hive database ,the libname code is

libname dm JDBC url="jdbc:hive2://xx.xx.xx.xx:10000/" classpath="D:\JDBC_Drivers_For_Hive" driverclass="com.cloudera.hive.jdbc.HS2Driver" 
user="*******" password="*********" schema="DM" DBMAX_TEXT=1024 PRESERVE_TAB_NAMES=YES PRESERVE_COL_NAMES=YES TRANSCODE_FAIL=WARNING QUOTE_CHAR='"' ;
libname mylib "d:\";
proc sql;
connect using DM;
create table mylib.the_big_datas 
select * from connection to DM
(
select * from dm.the_big_dat
); 
disconnect from DM; 
quit;

I try to select a big data from the hive database which has 1000 columns and 1.8 million rows. Here I got the error log as below:

ERROR: Fetch error: [Cloudera][HiveJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 30. Background thread ID: 31. Error caught: Java heap space.

ERROR: Close error: java.lang.OutOfMemoryError: Java heap space

what should I do to make the SQL query statements work ？

ANY suggestion is welcome！

Thanks in advance！

Re: when using libname jdbc to connect to hive an error appears

LinusH — Wed, 10 Jan 2024 10:47:38 GMT

Have you tried to do a smilar query from another tool (than SAS)?

I found this, not sure if it's applicable:

https://community.cloudera.com/t5/Support-Questions/Hive-Query-Error-java-lang-OutOfMemoryError-Java-heap-space/td-p/15028

topic when using libname jdbc to connect to hive an error appears in SAS Programming

when using libname jdbc to connect to hive an error appears

Re: when using libname jdbc to connect to hive an error appears