Can you overload the metadata server by running autoloads the surpass the 15 minute mark?

[TonyLMayo]

We customize the autoload process to have multiple autoloads (yes it is needed).

Each Autoload is separated by 15-second interval but, some of the autoloads can run in excess of the traditional 15 minute usually set. ( I think that is part of the issue).

I see (IN THE AUTOLOAD LOG) that each autoload spawns a separate SAS session where each session checks the metadata server for EVERY table in THAT autoload session to see if the table needs to be updated.

This is where I believe we push the metadata server to it's limits by all the calls for status.

Am I wrong?

should I beef up the metadata server?

lets say we have:

Autoload1

   5 tables

Autoload2 (delayed 15 sec)

   25 tables

Autoload3 (delayed 30 sec)

10 tables

there a call to the metadata server for each table which takes up resoures.

[SASKiwi]

IMHO, VA autoloading is best suited to occasional or ad-hoc end-user loading. If you have special requirements like high frequency or real-time loading then it is best to design a custom load process.

I suggest you consider a load process that is one SAS job and one session that just loops for the required number of times for a set period. For example you could have a load job that runs every hour and loops up to 240 times to give you the 15 second loading interval. That way you will avoid the overhead of starting a SAS session every 15 seconds. Also your loop logic could adjust for when it is fast or slow. By timing each loop you can get the job to sleep (SLEEP function) if the load has finished in less than 15 seconds, or just keep going if it exceeds 15 seconds. That way it will automatically self-correct.

[TonyLMayo]

I like it!!!