topic Re: Run Code on Only Newly imported Data in SAS Programming

Run Code on Only Newly imported Data

bklassen — Thu, 13 Sep 2018 17:33:47 GMT

Hi,

I am currently working on an automatic data refresh using Windows Task Scheduler. Every night at midnight web scraped data will be uploaded on my computer. I will then run the automated SAS code for data manipulation purposes. However I would only like to run the SAS code on the data that is new. My file size is very large, so I would like to reduce process time by only running the code through the new data rather than the entire data set. I was wondering if there is a code that would only run on new data.

Thank you!

Re: Run Code on Only Newly imported Data

Reeza — Thu, 13 Sep 2018 17:46:48 GMT

Sure, just add a step to your process so that it extracts the new data only and then runs on that. Exactly how depends a bit more on your data structure. There needs to be a relatively clean way to know what’s new data.

@bklassen wrote:

Hi,

I am currently working on an automatic data refresh using Windows Task Scheduler. Every night at midnight web scraped data will be uploaded on my computer. I will then run the automated SAS code for data manipulation purposes. However I would only like to run the SAS code on the data that is new. My file size is very large, so I would like to reduce process time by only running the code through the new data rather than the entire data set. I was wondering if there is a code that would only run on new data.

Thank you!

Re: Run Code on Only Newly imported Data

VDD — Thu, 13 Sep 2018 18:53:40 GMT

is there date and timestamp on the records? you could setup a step or selection statement for processing records using those indicators.