Run Code on Only Newly imported Data

[bklassen]

Hi,

I am currently working on an automatic data refresh using Windows Task Scheduler. Every night at midnight web scraped data will be uploaded on my computer. I will then run the automated SAS code for data manipulation purposes. However I would only like to run the SAS code on the data that is new. My file size is very large, so I would like to reduce process time by only running the code through the new data rather than the entire data set. I was wondering if there is a code that would only run on new data.

Thank you!

[Reeza]

Sure, just add a step to your process so that it extracts the new data only and then runs on that. Exactly how depends a bit more on your data structure. There needs to be a relatively clean way to know what’s new data. 

---

@bklassen wrote:

Hi,

I am currently working on an automatic data refresh using Windows Task Scheduler. Every night at midnight web scraped data will be uploaded on my computer. I will then run the automated SAS code for data manipulation purposes. However I would only like to run the SAS code on the data that is new. My file size is very large, so I would like to reduce process time by only running the code through the new data rather than the entire data set. I was wondering if there is a code that would only run on new data.

Thank you!

---

[VDD]

is there date and timestamp on the records?  you could setup a step or selection statement for processing records using those indicators.