CPU usage in the Viya 3.5 platform

[thedcoii]

We have a sas viya environment that was running very low CPU usage. However, under very little load the sas services would crash with an out of memory error. We increased the memsize setting and now the system doesn't crash but recently we've run into another issue with high cpu usage. 

It was running with only the memsize change for a week without a change in the cpu usage. In our last update however we added 3 new pickle files for models and updated some python libraries. The morning after this we saw very high cpu usage. 

Today we had more traffic then normal but not something that should overload the system. Still, we got to over 80% cpu usage and around the two times this happened we had users reporting slowness. 

So I really think it could only be three things. 

1.) Updated Python libraries causing issues

2.) The memsize causing cpu to be used differently

3.) Pickle files adding to the response times

Does anyone have ideas or where I should look to address these? 

[JuanS_OCS]

Hi there,

Considering the fact that 2 of your 3 options are related to Python and not to SAS, I would start at looking what exact system processes and or services are the ones going up in cpu.

There’s also the aspect of coincidence. Several things happening at the same time: try to filter them down and observe which one in fact causes the pattern of the problem.

In any case, do align with your SAS and system admins, and feel free to share (with us and/or SAS tech support) your further observations when you are still unable to get a conclusion on your own.

[thedcoii]

We identified it as a python library. Specifically, LightGBM 4.5.0 versus 2.3.1. In the later version the CPU usage by the python process in the linux box is much higher. It doesn't seem to be code related either as the older versions that only use the features of 2.3.1 appear to exhibit this behavior as well. However, that could be misleading if the higher usage ones are all tied to 4.5.0