BookmarkSubscribeRSS Feed
duanzongran
Obsidian | Level 7

dear all:

I use jdbc to connect to hive database ,the libname code is

libname dm JDBC url="jdbc:hive2://xx.xx.xx.xx:10000/" classpath="D:\JDBC_Drivers_For_Hive" driverclass="com.cloudera.hive.jdbc.HS2Driver" 
user="*******" password="*********" schema="DM" DBMAX_TEXT=1024 PRESERVE_TAB_NAMES=YES PRESERVE_COL_NAMES=YES TRANSCODE_FAIL=WARNING QUOTE_CHAR='"' ;
libname mylib "d:\";
proc sql;
connect using DM;
create table mylib.the_big_datas
select * from connection to DM
(
select * from dm.the_big_dat
);
disconnect from DM;
quit;

I try to select a big data  from the hive database  which has 1000 columns and 1.8 million rows. Here  I got the error log as below:

ERROR: Fetch error: [Cloudera][HiveJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 30. Background thread ID: 31. Error caught: Java heap space.

ERROR: Close error: java.lang.OutOfMemoryError: Java heap space

what should I do to make the SQL query statements work ?

ANY suggestion is welcome!

Thanks in advance!

 

 

1 REPLY 1
LinusH
Tourmaline | Level 20

Have you tried to do a smilar query from another tool (than SAS)?

I found this, not sure if it's applicable:

https://community.cloudera.com/t5/Support-Questions/Hive-Query-Error-java-lang-OutOfMemoryError-Java...

Data never sleeps

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 1 reply
  • 422 views
  • 0 likes
  • 2 in conversation