Log fragmentation - new log for each event between 00:00 and 01:00

[AgnetheSO]

Hi, 

We are working on installing a test environment. The logs for our object spawners and the metadata server have started splitting up between 00:00 and 01:00. Instead of having one ".log" file for each day, we now have a log file for most of the day and about 7000 log files created between 00:00 and 01:00. 

Each log file contains one event, f.ex. 

Host: 'COLO-SASMETA', OS: 'WIN', Release: '6.3', SAS Version: '9.04.01M5P09132017', Command: '"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -config "C:\SAS\Config\Lev1\SASMeta\MetadataServer\sasv9.cfg" -servicename "SAS [Config-Lev1] SASMeta - Metadata Server"'
Log continued from C:\SAS\Config\Lev1\SASMeta\MetadataServer\Logs\SASMeta_MetadataServer_2019-10-03_COLO-SASMETA_4268.log_366
2019-10-03T00:57:24,434 INFO [06027745] :sasevs@saspw - New client connection (2139943) accepted from server port 8561 for user sasevs@saspw. Encryption level is Credentials using encryption algorithm AES. Peer IP address and port are [::ffff:10.200.2.233]:65345 for APPNAME=SAS Object Spawner.
Log continues in C:\SAS\Config\Lev1\SASMeta\MetadataServer\Logs\SASMeta_MetadataServer_2019-10-03_COLO-SASMETA_4268.log_368

I am quite new to SAS installation work, and can't figure out why this is happening.

Has anyone seen this behaviour before?

What could cause time limited log fragmentation?

Thanks,

A

[Anand_V]

Hi @AgnetheSO ,

If you have any similar environment, first thing I would suggest is to compare the logconfig.xml file for Metadata server and Object spawner and see if there are any differences.

Example path:

C:\SAS\Config\Lev1\SASMeta\MetadataServer\logconfig.xml

Thanks!

[AgnetheSO]

Hi @Anand_V , 

I probably should have mentioned this in my original post, but the log configuration files are identical across the environments. 

Thanks 

[AgnetheSO]

Additional information:

The test platform is installed across four machines (1 metadata, 2 compute, 1 web tier).

I've had a look in Environment Manager and a couple of other things are happening in the 00:00 - 01:00 time period.

For all machines:

-  30-40 new processes start up and then stop

- There is a dip in available memory 

- There is an increase in cpu usage

- There is an increase in both inbound and outbound TCP connections

- There are more 'TCP Attempt Fails' alerts fired

On the compute servers, for the object spawners:

- 00:00 - Object Spawner Server Health is 0.0% for the object spawners that are not load balanced

- There is then a peak in Process Page Faults per Minute and Process Threads

- There is a dip in Process Resident Memory Size

- 01:00 - Object Spawner Server Health is 0.0% for the object spawners that are load balanced