BookmarkSubscribeRSS Feed
mleitson
Calcite | Level 5

Hello All,

 

We are attempting to use SAS DIS 4.902 to load Oracle tables into Hadoop using the 'SAS Data in HDFS Loader' (which will in turn be loaded into the LASR serve via 'SAS LASR Analytic Server Loader').

 

This is efficient for most tables, but for a few tables, this process takes 3+ hours. We have been searching for ways to reduce the loading time, but have not found anything that works. Can anyone recommend a faster method to load Oracle tables into Hadoop? We know that billion dollar MNC’s are globally federating data into centralized warehouses daily in a relatively short time, so this has to be possible.

 

Thanks,

ML

3 REPLIES 3
AhmedAl_Attar
Rhodochrosite | Level 12

Hi @mleitson,

 

Not knowing your Network topology, ie. where Your SAS Workspace Server located in contrast to Oracle and Hadoop/HDFS nodes

You'll need to make sure your SAS-ORACLE connection is optimized to read higher volumes of records per fetch operation.

The default is 250 record per fetch, Check this link LIBNAME Statement Specifics for Oracle, and look at the BUFFSIZE= option.

 

Increasing the value associated with the option, can impact the amount of memory allocated to your SAS Workspace Server session (-MEMSIZE option), therefore I would look into increasing the -MEMSIZE setting first, before adjusting the BUFFSIZE= value.

 

The other option to speed the Loading Process, is Running Multiple concurrent SAS jobs, which extracts Oracle Records based on specific ROWID ranges, and try to load them into Hadoop HDFS in parallel.

 

Hope this help,

Ahmed  

 

ccaulkins9
Pyrite | Level 9
Leitson,

Did you try cross-posting to this group:
SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop, SAS Data Preparation and others
? @AhmedAl_Attar
e-SAS regards,

ccaulkins9
Pyrite | Level 9

@mleitson 

e-SAS regards,

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

Mastering the WHERE Clause in PROC SQL

SAS' Charu Shankar shares her PROC SQL expertise by showing you how to master the WHERE clause using real winter weather data.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 1022 views
  • 5 likes
  • 3 in conversation