Architecting, installing and maintaining your SAS environment

Best way for frequent and fast refresh of data on LASR server

Reply
Occasional Contributor
Posts: 9

Best way for frequent and fast refresh of data on LASR server

Hi,
I use SAS VA 7.4

It is set on sas base 9.4 and of course the LASR version is 2.8

The system look like this:
- every hour, the data is refreshed on other servers in different systems. I created view to have access to them

The question is:
What is the most efficient way to update my data?

Requirements:
It has to update very frequently (even during the day) and the proces must be very short.
The data must be up to date the moment it changes.

My thoughts:
I thought of:
- using IMSTAT and doing all operations in memory
- or just doing it in tables, hash tables and then update the memory
I receive data like which rows are to be removed and which to append

Super User
Posts: 3,252

Re: Best way for frequent and fast refresh of data on LASR server

Posted in reply to jakubniescior

How big are your tables? Do you want to refresh completely or just the rows that have changed? What percentage of rows change between updates? Have you tried a loading test and how long does it take?

Occasional Contributor
Posts: 9

Re: Best way for frequent and fast refresh of data on LASR server

Thank you SASKiwi for your prompt reply.

 

My main table has to contain about 22 000 000 of rows of data with about 20 columns.

 

Every hour, there will be removed ~1000 of rows and appended another ~1000 rows,

what makes it less than 1 %.

 

I did not try any loading tests and honestly I can't think of any for now.

I did try to do deleting with IMSTAT, but for some reason, deleting ~90000 rows from table with 200 000 rows took me more than 10 minutes...

Super User
Posts: 3,252

Re: Best way for frequent and fast refresh of data on LASR server

Posted in reply to jakubniescior

I agree that removing and appending with IMSTAT is the way to go. What is performance like removing and adding 1,000 rows?

Ask a Question
Discussion stats
  • 3 replies
  • 237 views
  • 0 likes
  • 2 in conversation