SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

can merge transformation be applied to hive table?

Reply
Regular Contributor
Posts: 181

can merge transformation be applied to hive table?

Hi,

 

Can a merge transformation be applied to a hive table using a sas dataset?

 

Appreciate any help.

 

Regards,
Sheeba

Super User
Posts: 5,849

Re: can merge transformation be applied to hive table?

Yes, but the merge operation will take place in SAS, not in Hive.
Data never sleeps
Regular Contributor
Posts: 181

Re: can merge transformation be applied to hive table?

Hi Linush,

 

Thanks for the reply.

 

I believe it wouldnt impact the data right.

 

I am looking for  inserting new rows and updating existing rows which is same as what merge transformation does. i Have used it to update sas datasets but i have not tried this to update hive tables.. Is there any other way to do an insert and update to a hive table using sas DI studio?

 

Regards,

Sheeba

Super User
Posts: 5,849

Re: can merge transformation be applied to hive table?

data step merge does not update/insert data at rest, it's a type of query that results in a new data set.

So from a data management/ETL perspective I would say this is a no-no.

There are ready made transformations for dealing with target table updates, select the one appropriate to your logic.

Sounds like that you need help from an ETL designer...?

Data never sleeps
Regular Contributor
Posts: 181

Re: can merge transformation be applied to hive table?

Hi Linush,

 

tnx a lot for the details.

 

I will look into the transformations which does the target table updates.

 

Regards,

Sheeba

Super User
Posts: 5,849

Re: can merge transformation be applied to hive table?

Bare in mind that Hive have limited set of methods for update (none until recently). So be sure to align your ETL with Hive capabilities.

Data never sleeps
Regular Contributor
Posts: 181

Re: can merge transformation be applied to hive table?

Hi Linush,

 

Thanks. Sure I will check if it is supported by SAS DI.

 

Regards,

Sheeba

Respected Advisor
Posts: 4,672

Re: can merge transformation be applied to hive table?

[ Edited ]

@Sheeba

Given that Hadoop is only good at Insert (Append) and Read you might also consider a strategy as outlined here:

https://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/  

Regular Contributor
Posts: 181

Re: can merge transformation be applied to hive table?

Hi Patrick,

 

Thanks a lot for the link. It is really helpful.

 

I will try to deep dive into it and follow it.

Regards,

Sheeba

Ask a Question
Discussion stats
  • 8 replies
  • 238 views
  • 4 likes
  • 3 in conversation