BookmarkSubscribeRSS Feed
jakubniescior
Fluorite | Level 6

Hi,
I use SAS VA 7.4

It is set on sas base 9.4 and of course the LASR version is 2.8

The system look like this:
- every hour, the data is refreshed on other servers in different systems. I created view to have access to them

The question is:
What is the most efficient way to update my data?

Requirements:
It has to update very frequently (even during the day) and the proces must be very short.
The data must be up to date the moment it changes.

My thoughts:
I thought of:
- using IMSTAT and doing all operations in memory
- or just doing it in tables, hash tables and then update the memory
I receive data like which rows are to be removed and which to append

3 REPLIES 3
SASKiwi
PROC Star

How big are your tables? Do you want to refresh completely or just the rows that have changed? What percentage of rows change between updates? Have you tried a loading test and how long does it take?

jakubniescior
Fluorite | Level 6

Thank you SASKiwi for your prompt reply.

 

My main table has to contain about 22 000 000 of rows of data with about 20 columns.

 

Every hour, there will be removed ~1000 of rows and appended another ~1000 rows,

what makes it less than 1 %.

 

I did not try any loading tests and honestly I can't think of any for now.

I did try to do deleting with IMSTAT, but for some reason, deleting ~90000 rows from table with 200 000 rows took me more than 10 minutes...

SASKiwi
PROC Star

I agree that removing and appending with IMSTAT is the way to go. What is performance like removing and adding 1,000 rows?

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

CLI in SAS Viya

Learn how to install the SAS Viya CLI and a few commands you may find useful in this video by SAS’ Darrell Barton.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 959 views
  • 0 likes
  • 2 in conversation