07-13-2017 10:59 AM
I use SAS VA 7.4
It is set on sas base 9.4 and of course the LASR version is 2.8
The system look like this:
- every hour, the data is refreshed on other servers in different systems. I created view to have access to them
The question is:
What is the most efficient way to update my data?
It has to update very frequently (even during the day) and the proces must be very short.
The data must be up to date the moment it changes.
I thought of:
- using IMSTAT and doing all operations in memory
- or just doing it in tables, hash tables and then update the memory
I receive data like which rows are to be removed and which to append
07-14-2017 03:38 AM
How big are your tables? Do you want to refresh completely or just the rows that have changed? What percentage of rows change between updates? Have you tried a loading test and how long does it take?
07-14-2017 04:30 AM
Thank you SASKiwi for your prompt reply.
My main table has to contain about 22 000 000 of rows of data with about 20 columns.
Every hour, there will be removed ~1000 of rows and appended another ~1000 rows,
what makes it less than 1 %.
I did not try any loading tests and honestly I can't think of any for now.
I did try to do deleting with IMSTAT, but for some reason, deleting ~90000 rows from table with 200 000 rows took me more than 10 minutes...
07-27-2017 03:59 PM
I agree that removing and appending with IMSTAT is the way to go. What is performance like removing and adding 1,000 rows?