High CPU usage by Java causes Create Table to Fail

[MarkESmith]

Hey all,

Recently noticed an issue with the SAS machine (Running mainly Visual Analytics - 7.3, non-distributed mode, on Red Hat Enterprise Linux 7, 180 GB of physical RAM) that, at this point, seems to be specifically a problem for about an hour or two after rebooting. After rebooting the SAS machine, starting the SAS servers (sas.servers script) and starting the LASR server - we have queries set up to load tables automatically.

Unfortunately, last time these tables ran, most of the tables did not load successfully (in this instance, the LASR server was started by Autoload). When looking at the 'Last Action Log', I see the following:

-1
-------------- Task Summary --------------
ERRORS (Create Query and Load)
----------------------------------------------
ERROR: Insufficient resources to perform the operation.
ERROR: The SAS System stopped processing this step because of insufficient memory.

I decided to reboot again (before doing so, I turned off AutoLoad.AutoStart for the LASR server), I started the LASR server manually, and I tried to load the tables manually instead of automatically (my first thought was that perhaps having the queries run automatically was too taxing on memory because of the small window in which we start all the queries). But I found that for many of the tables, I was receiving the same error (the much smaller tables appeared to load successfully) that occurred with the automatic loading of the tables. Granted, I usually load about 3 or 4 of these tables at at a time. As a result of these errors occurring, I had to load each table, one-by-one, waiting for each one to finish before beginning the next. 

While I was waiting for the tables to load I decided to check out 'top' in an SSH terminal. It appeared to me that a particular Java process was taking up large amounts of of CPU (and virtual memory). This made me think that this process was a likely suspect for the cause of these 'insufficient resources' errors.I wanted to know if anyone else has had this problem or a problem similar: Here's a screenshot of what I was seeing in regards to the CPU/Memory usage of Java. This process consistently took up large amounts of memory and CPU for about 2.5-3 hours, after which it seem to go down and no more of those errors popped up when loading tables.

Does anyone have any ideas if this is normal and, if not, why a Java process may be behaving like this? Any tips or ideas of what log files to look at, configurations to check, etc. would be much appreciated.

Thanks in advance!!

[Anand_V]

Hi @MarkESmith ,

Has the auto-load work previously? Did you try the steps in this KB? http://support.sas.com/kb/53/316.html

Thanks!

[MarkESmith]

It works in so far as AutoLoad starts the LASR server - but as I noted in the original post, even when I started the LASR server manually and manually reloaded the tables, I ran into those error messages.

I have verified that my settings are correct per the KB. There is one discrepancy though. See below:

From the KB:

USERMODS_OPTIONS="-memsize 0 -cpucount ACTUAL"

Mine:

USERMODS_OPTIONS="-memsize 0 -cpucount ACTUAL "

Would that extra space between the 'L' and the closing " make any difference? (If it did I would be quite surprised).