BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
89sankar
Fluorite | Level 6

When I use Table loader append to existing data, I get the latest data appended to old data. While doing this, incase of error of space issue, how to avoid appending again the latest data two times?

1 ACCEPTED SOLUTION

Accepted Solutions
Patrick
Opal | Level 21

You could use Update/Insert - but this will slow down performance.

 

One of the advantages in using DIS and scheduling is more control over your processes. Executing the same job twice on the same data shouldn't happen (eg. once data loaded copy the data source to an archive directory as part of your process).

 

You could also maintain some sort of control table where you keep the max. date of data loaded; and you then use this max date in a where clause to only select newer dates from your source. Then have a post process which updates the control table with the latest max. date.

View solution in original post

6 REPLIES 6
Reeza
Super User

There's no automatic way, assuming you're in base SAS. You need to define some rules on how to check if the process should proceed. 

89sankar
Fluorite | Level 6

Do we have any option to do this setting in table loader? to make sure that I am not appending the latest data again?

Reeza
Super User

Are you referring to Data Integration Studio? Or DaraFlux!

89sankar
Fluorite | Level 6

Data integration studio - Table loader transformation

Patrick
Opal | Level 21

You could use Update/Insert - but this will slow down performance.

 

One of the advantages in using DIS and scheduling is more control over your processes. Executing the same job twice on the same data shouldn't happen (eg. once data loaded copy the data source to an archive directory as part of your process).

 

You could also maintain some sort of control table where you keep the max. date of data loaded; and you then use this max date in a where clause to only select newer dates from your source. Then have a post process which updates the control table with the latest max. date.

89sankar
Fluorite | Level 6

Thanks!! It helps.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 6 replies
  • 1690 views
  • 1 like
  • 3 in conversation