SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

Table loader append to exsting

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 10
Accepted Solution

Table loader append to exsting

When I use Table loader append to existing data, I get the latest data appended to old data. While doing this, incase of error of space issue, how to avoid appending again the latest data two times?


Accepted Solutions
Solution
‎11-27-2016 11:17 PM
Respected Advisor
Posts: 3,891

Re: Table loader append to exsting

[ Edited ]

You could use Update/Insert - but this will slow down performance.

 

One of the advantages in using DIS and scheduling is more control over your processes. Executing the same job twice on the same data shouldn't happen (eg. once data loaded copy the data source to an archive directory as part of your process).

 

You could also maintain some sort of control table where you keep the max. date of data loaded; and you then use this max date in a where clause to only select newer dates from your source. Then have a post process which updates the control table with the latest max. date.

View solution in original post


All Replies
Super User
Posts: 17,828

Re: Table loader append to exsting

There's no automatic way, assuming you're in base SAS. You need to define some rules on how to check if the process should proceed. 

Occasional Contributor
Posts: 10

Re: Table loader append to exsting

Do we have any option to do this setting in table loader? to make sure that I am not appending the latest data again?

Super User
Posts: 17,828

Re: Table loader append to exsting

Are you referring to Data Integration Studio? Or DaraFlux!

Occasional Contributor
Posts: 10

Re: Table loader append to exsting

Data integration studio - Table loader transformation

Solution
‎11-27-2016 11:17 PM
Respected Advisor
Posts: 3,891

Re: Table loader append to exsting

[ Edited ]

You could use Update/Insert - but this will slow down performance.

 

One of the advantages in using DIS and scheduling is more control over your processes. Executing the same job twice on the same data shouldn't happen (eg. once data loaded copy the data source to an archive directory as part of your process).

 

You could also maintain some sort of control table where you keep the max. date of data loaded; and you then use this max date in a where clause to only select newer dates from your source. Then have a post process which updates the control table with the latest max. date.

Occasional Contributor
Posts: 10

Re: Table loader append to exsting

Thanks!! It helps.

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 6 replies
  • 271 views
  • 1 like
  • 3 in conversation