topic Re: Issue while loading data into Hadoop from Oracle in New SAS User

Issue while loading data into Hadoop from Oracle

india2016 — Tue, 09 Oct 2018 07:48:24 GMT

Hello Team,

We are facing issue while loading data into hadoop environment from Oracle database using DI job (Table Loader Node).
But when table is loaded into hadoop , multiple missing observations are generated which are not in Oracle data-base.

Anyone faced this issue then please suggest what should I do to overcome this issue?

Any suggestion will be appreciated.

Re: Issue while loading data into Hadoop from Oracle

JBailey — Wed, 10 Oct 2018 15:26:53 GMT

Hi @india2016

Without detailed information it is hard to answer your question.

If it were me, I would validate the data coming out of Oracle and see if there are any surprises. Once I am sure the Oracle data looks good I would load it into Hadoop. Pay attention to the data type conversions and missing values. If you can find the Oracle rows that display the problems it would help the community members understand what is happening.

To really solve this we would need, at a minimum, the DDL for the table definitions in both Oracle and Hive and sample data (that causes the problem), and data (from Hive) showing the problem.

Best wishes,

Jeff

Re: Issue while loading data into Hadoop from Oracle

india2016 — Thu, 11 Oct 2018 05:10:35 GMT

1) When we run a DI job it fetches data from Oracle table and loads it into Hadoop table using Table loader transformation.

The data gets loaded into Hadoop Table and then when you open the table in SAS environment you are getting some blank rows.

2) As you suggested, there is no such blank values in Oracle table but while data loading into Hadoop dataset,some extra missing rows were generated in hadoop table.

DDL is created in Hadoop using CREATE TABLE node. and then using DATA LOADER node data was loaded into Hadoop tables.

Should I share backend code for these transformations?

I have attached backend coed for the same.

Please find attached log.