We customize the autoload process to have multiple autoloads (yes it is needed).
Each Autoload is separated by 15-second interval but, some of the autoloads can run in excess of the traditional 15 minute usually set. ( I think that is part of the issue).
I see (IN THE AUTOLOAD LOG) that each autoload spawns a separate SAS session where each session checks the metadata server for EVERY table in THAT autoload session to see if the table needs to be updated.
This is where I believe we push the metadata server to it's limits by all the calls for status.
Am I wrong?
should I beef up the metadata server?
lets say we have:
Autoload2 (delayed 15 sec)
Autoload3 (delayed 30 sec)
there a call to the metadata server for each table which takes up resoures.
IMHO, VA autoloading is best suited to occasional or ad-hoc end-user loading. If you have special requirements like high frequency or real-time loading then it is best to design a custom load process.
I suggest you consider a load process that is one SAS job and one session that just loops for the required number of times for a set period. For example you could have a load job that runs every hour and loops up to 240 times to give you the 15 second loading interval. That way you will avoid the overhead of starting a SAS session every 15 seconds. Also your loop logic could adjust for when it is fast or slow. By timing each loop you can get the job to sleep (SLEEP function) if the load has finished in less than 15 seconds, or just keep going if it exceeds 15 seconds. That way it will automatically self-correct.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.