topic Best way for frequent and fast refresh of data on LASR server in Administration and Deployment

Best way for frequent and fast refresh of data on LASR server

jakubniescior — Thu, 13 Jul 2017 14:59:59 GMT

Hi,
I use SAS VA 7.4

It is set on sas base 9.4 and of course the LASR version is 2.8

The system look like this:
- every hour, the data is refreshed on other servers in different systems. I created view to have access to them

The question is:
What is the most efficient way to update my data?

Requirements:
It has to update very frequently (even during the day) and the proces must be very short.
The data must be up to date the moment it changes.

My thoughts:
I thought of:
- using IMSTAT and doing all operations in memory
- or just doing it in tables, hash tables and then update the memory
I receive data like which rows are to be removed and which to append

Re: Best way for frequent and fast refresh of data on LASR server

SASKiwi — Fri, 14 Jul 2017 07:38:00 GMT

How big are your tables? Do you want to refresh completely or just the rows that have changed? What percentage of rows change between updates? Have you tried a loading test and how long does it take?

Re: Best way for frequent and fast refresh of data on LASR server

jakubniescior — Fri, 14 Jul 2017 08:30:49 GMT

Thank you SASKiwi for your prompt reply.

My main table has to contain about 22 000 000 of rows of data with about 20 columns.

Every hour, there will be removed ~1000 of rows and appended another ~1000 rows,

what makes it less than 1 %.

I did not try any loading tests and honestly I can't think of any for now.

I did try to do deleting with IMSTAT, but for some reason, deleting ~90000 rows from table with 200 000 rows took me more than 10 minutes...

Re: Best way for frequent and fast refresh of data on LASR server

SASKiwi — Thu, 27 Jul 2017 19:59:20 GMT

I agree that removing and appending with IMSTAT is the way to go. What is performance like removing and adding 1,000 rows?