BookmarkSubscribeRSS Feed
Sheeba
Lapis Lazuli | Level 10

Hi,

 

Can a merge transformation be applied to a hive table using a sas dataset?

 

Appreciate any help.

 

Regards,
Sheeba

8 REPLIES 8
LinusH
Tourmaline | Level 20
Yes, but the merge operation will take place in SAS, not in Hive.
Data never sleeps
Sheeba
Lapis Lazuli | Level 10

Hi Linush,

 

Thanks for the reply.

 

I believe it wouldnt impact the data right.

 

I am looking for  inserting new rows and updating existing rows which is same as what merge transformation does. i Have used it to update sas datasets but i have not tried this to update hive tables.. Is there any other way to do an insert and update to a hive table using sas DI studio?

 

Regards,

Sheeba

LinusH
Tourmaline | Level 20

data step merge does not update/insert data at rest, it's a type of query that results in a new data set.

So from a data management/ETL perspective I would say this is a no-no.

There are ready made transformations for dealing with target table updates, select the one appropriate to your logic.

Sounds like that you need help from an ETL designer...?

Data never sleeps
Sheeba
Lapis Lazuli | Level 10

Hi Linush,

 

tnx a lot for the details.

 

I will look into the transformations which does the target table updates.

 

Regards,

Sheeba

LinusH
Tourmaline | Level 20

Bare in mind that Hive have limited set of methods for update (none until recently). So be sure to align your ETL with Hive capabilities.

Data never sleeps
Sheeba
Lapis Lazuli | Level 10

Hi Linush,

 

Thanks. Sure I will check if it is supported by SAS DI.

 

Regards,

Sheeba

Patrick
Opal | Level 21

@Sheeba

Given that Hadoop is only good at Insert (Append) and Read you might also consider a strategy as outlined here:

https://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/  

Sheeba
Lapis Lazuli | Level 10

Hi Patrick,

 

Thanks a lot for the link. It is really helpful.

 

I will try to deep dive into it and follow it.

Regards,

Sheeba

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 8 replies
  • 1257 views
  • 4 likes
  • 3 in conversation