Viya 3.5 CAS Controller service down frequently

[saiprasad_fino]

We are facing CAS controller service down issue frequently. We checked logs on Controller & workers also.

Checked the below mentioned logs and suggest the appropriate solution.

2022-07-26T13:52:37,662 INFO [03821821] MAIN NoUser MAIN [tkcsesinst.c:2160] - User sas.searchIndex attached to session 79b5f799-480c-764a-86f5-38288d51edb9.
2022-07-26T13:52:37,720 INFO [00000007] 724065 sas.searchIndex 168143 [tkcasaimp.c:4504] - ++ action table.caslibInfo / srcType='all', verbose=true, showHidden=true;
2022-07-26T13:52:37,722 INFO [00000007] 724065 sas.searchIndex 168143 [tkcasaimp.c:6460] - -- 'table.caslibInfo' SUCCESSFUL.
2022-07-26T13:52:37,786 INFO [03821822] MAIN NoUser MAIN [tkcalsident.c:424] - New client connection accepted on port 5570. Client IP address and port are [10.71.87.54]:47724. TLS is required.
2022-07-26T13:52:37,786 INFO [03821822] MAIN NoUser MAIN [tkident.c:1368] - User sas.searchIndex successfully authenticated using the OAuth authentication provider.
2022-07-26T13:52:37,786 INFO [03821822] MAIN NoUser MAIN [tkcsesinst.c:2160] - User sas.searchIndex attached to session 79b5f799-480c-764a-86f5-38288d51edb9.
2022-07-26T13:52:37,842 INFO [00000007] 724065 sas.searchIndex 168143 [tkcasaimp.c:4504] - ++ action session.listSessions;
2022-07-26T13:52:37,844 INFO [00000007] 724065 sas.searchIndex 168143 [tkcasaimp.c:6460] - -- 'session.listSessions' SUCCESSFUL.
2022-07-26T13:52:37,892 INFO [03821823] MAIN NoUser MAIN [tkcalsident.c:424] - New client connection accepted on port 5570. Client IP address and port are [10.71.87.54]:47730. TLS is required.
2022-07-26T13:52:37,893 INFO [03821823] MAIN NoUser MAIN [tkident.c:1368] - User sas.searchIndex successfully authenticated using the OAuth authentication provider.
2022-07-26T13:52:37,893 INFO [03821823] MAIN NoUser MAIN [tkcsesinst.c:2160] - User sas.searchIndex attached to session 79b5f799-480c-764a-86f5-38288d51edb9.
2022-07-26T13:52:37,949 INFO [00000007] 724065 sas.searchIndex 168143 [tkcasaimp.c:4504] - ++ action session.endSession;
2022-07-26T13:52:37,951 INFO [00000007] 724065 sas.searchIndex 168143 [tkcasaimp.c:6460] - -- 'session.endSession' SUCCESSFUL.
2022-07-26T13:52:37,952 INFO [00000007] 724065 sas.searchIndex 168143 [tkcsesinst.c:835] - Destroying session 79b5f799-480c-764a-86f5-38288d51edb9.
2022-07-26T13:52:38,067 INFO [03821818] MAIN sas.searchIndex 168143 [tkcsesinst.c:835] - Destroying session 79b5f799-480c-764a-86f5-38288d51edb9.
2022-07-26T13:52:41,336 INFO [00000007] 685786 102565 167971 [tkcasaimp.c:4504] - ++ action session.endSession;
2022-07-26T13:52:41,338 INFO [00000007] 685786 102565 167971 [tkcasaimp.c:6460] - -- 'session.endSession' SUCCESSFUL.
2022-07-26T13:52:41,339 INFO [00000007] 685786 102565 167971 [tkcsesinst.c:835] - Destroying session a9638555-cbda-8548-afa5-e44bf6d63437.
2022-07-26T13:52:41,390 INFO [00000007] 691182 102565 167986 [tkcasaimp.c:4504] - ++ action session.endSession;
2022-07-26T13:52:41,391 INFO [00000007] 691182 102565 167986 [tkcasaimp.c:6460] - -- 'session.endSession' SUCCESSFUL.
2022-07-26T13:52:41,392 INFO [00000007] 691182 102565 167986 [tkcsesinst.c:835] - Destroying session 7e9c973f-933f-3741-b608-379818e3c623.
2022-07-26T13:52:41,401 INFO [00000007] 644494 112064 167767 [tkcasaimp.c:4504] - ++ action session.endSession;
2022-07-26T13:52:41,402 INFO [00000007] 644494 112064 167767 [tkcasaimp.c:6460] - -- 'session.endSession' SUCCESSFUL.
2022-07-26T13:52:41,403 INFO [00000007] 644494 112064 167767 [tkcsesinst.c:835] - Destroying session 583b06d0-f39b-544d-bfb9-329d6dc10e94.
2022-07-26T13:52:41,444 INFO [00000007] 643995 112064 167760 [tkcasaimp.c:4504] - ++ action session.endSession;
2022-07-26T13:52:41,446 INFO [00000007] 643995 112064 167760 [tkcasaimp.c:6460] - -- 'session.endSession' SUCCESSFUL.
2022-07-26T13:52:41,447 INFO [00000007] 643995 112064 167760 [tkcsesinst.c:835] - Destroying session 4ac46ff5-5fbf-ef49-9706-69045e062aef.
2022-07-26T13:52:41,502 INFO [03809207] MAIN 112064 167760 [tkcsesinst.c:835] - Destroying session 4ac46ff5-5fbf-ef49-9706-69045e062aef.
2022-07-26T13:52:41,519 INFO [03816007] MAIN 102565 167971 [tkcsesinst.c:835] - Destroying session a9638555-cbda-8548-afa5-e44bf6d63437.
2022-07-26T13:52:41,524 INFO [03816663] MAIN 102565 167986 [tkcsesinst.c:835] - Destroying session 7e9c973f-933f-3741-b608-379818e3c623.
2022-07-26T13:52:41,586 INFO [03809227] MAIN 112064 167767 [tkcsesinst.c:835] - Destroying session 583b06d0-f39b-544d-bfb9-329d6dc10e94.
2022-07-26T13:52:54,976 ERROR [03821925] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:52:54,976 ERROR [03821925] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:52:55,064 ERROR [03821911] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:52:55,064 ERROR [03821911] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:52:55,103 ERROR [03821931] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:52:55,103 ERROR [03821931] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:52:55,103 ERROR [03821912] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:52:55,103 ERROR [03821912] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:52:55,125 ERROR [03821916] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:52:55,125 ERROR [03821916] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:54:03,712 ERROR [03822247] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:54:03,712 ERROR [03822247] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:54:03,769 ERROR [03822258] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:54:03,769 ERROR [03822258] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:54:03,793 ERROR [03822253] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:54:03,793 ERROR [03822253] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:54:03,803 ERROR [03822240] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:54:03,803 ERROR [03822240] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error
2022-07-26T13:54:03,824 ERROR [03822265] MAIN NoUser MAIN [httpscomm.c:597] - Error reading HTTP header from client.
2022-07-26T13:54:03,824 ERROR [03822265] MAIN NoUser MAIN [httpscomm.c:598] - Encryption run-time execution error

[gwootton]

Those errors usually mean that a client is connecting to the CAS port but not communicating in the way CAS expects, often a port scanner might result in these messages. Do these errors coincide with your CAS server going down?

[saiprasad_fino]

Thanks for the reply. Whenever CAS controller service going down, at that time log showing these errors and alerts. Our CAS controller service going down frequently from last 2-3 days and need to restart manually. We checked logs from Controller and worker nodes also but showing same logs as I mentioned in previous mail. Please help that why it is happening frequently...

[gwootton]

These log messages are the result of some external process contacting CAS incorrectly. For example, trying to establish a binary connection to the CAS server on the HTTP port (8777) instead of the binary port (5570), or a port scanner just trying to connect on every port on a given server to see what ports are open for security testing. Do you see the errors in places where the downtime did not occur?

If these are causing your CAS server to go down, you should identify where those errant connections are coming from and put a stop to them. You could probably turn up the logging on the CAS server to get some more information on where those connections are coming from.

That said, I'm not sure these connections are the cause of the downtime, so you may wish to engage SAS Technical Support.