Loading large tables into Hadoop

[mleitson]

Hello All,

We are attempting to use SAS DIS 4.902 to load Oracle tables into Hadoop using the 'SAS Data in HDFS Loader' (which will in turn be loaded into the LASR serve via 'SAS LASR Analytic Server Loader').

This is efficient for most tables, but for a few tables, this process takes 3+ hours. We have been searching for ways to reduce the loading time, but have not found anything that works. Can anyone recommend a faster method to load Oracle tables into Hadoop? We know that billion dollar MNC’s are globally federating data into centralized warehouses daily in a relatively short time, so this has to be possible.

Thanks,

ML

[AhmedAl_Attar]

Hi @mleitson,

Not knowing your Network topology, ie. where Your SAS Workspace Server located in contrast to Oracle and Hadoop/HDFS nodes

You'll need to make sure your SAS-ORACLE connection is optimized to read higher volumes of records per fetch operation.

The default is 250 record per fetch, Check this link LIBNAME Statement Specifics for Oracle, and look at the BUFFSIZE= option.

Increasing the value associated with the option, can impact the amount of memory allocated to your SAS Workspace Server session (-MEMSIZE option), therefore I would look into increasing the -MEMSIZE setting first, before adjusting the BUFFSIZE= value.

The other option to speed the Loading Process, is Running Multiple concurrent SAS jobs, which extracts Oracle Records based on specific ROWID ranges, and try to load them into Hadoop HDFS in parallel.

Hope this help,

Ahmed  

[ccaulkins9]

Leitson,

Did you try cross-posting to this group:
SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop, SAS Data Preparation and others
? @AhmedAl_Attar

[ccaulkins9]

@mleitson