Reference Name: Security Software disabling SAS Applications
Severity: High
Status: New and Improved Third-Party Security and System Monitoring Applications are constantly being introduced and evolving. As they do, their configuration and activities can interfere with, or stop SAS applications from executing. This is an ongoing problem and is tracked continuously. Periodically review this note for changes or additions, and advice.
Impact
SAS Foundation Applications like LASR, base SAS, SAS GRID,; and SAS End User Applications like VA, VI, VS, etc. can be hampered from performing well, or completely denied resources or permission to run by the configurations and activities of Third-Party Security or Monitoring Software. This can completely disable the SAS Application, resulting in business stoppage.
Description
A recently high number of SAS Users engaged SAS Technical Support to discover why their SAS applications/jobs were not running. Most of the situations involved long-standing, existing systems, which had recently upgraded their OS, Security Software, and/or added System Monitoring. The systems had added or made changes to Endpoint Security Protection or Monitoring Software:
Endpoint Security Protection Software
Endpoint security monitors and protects endpoints (workstations, servers, phones, tablets, networks, switches, etc.) that can be the entry point of malware, cyberattacks, and other security invasions. It allows organizations to protect their computer networks, networks, and data for hackers and malware. In securing these endpoints, the solution actively monitors, detects threats, investigates, and responds to unsafe or suspicious behavior. This can include activities like virus checking and containment, malware interception, preventing applications from using resources, or stopping application executions that fall within pre-configured protocols.
Given the proliferation and adoption of Endpoint Security in todays fast moving security market, we have had 6 customers in the last two months have their applications disabled by Endpoint Protection software they installed.
The most frequent Endpoint Protection Applications encountered stopping SAS Applications from running have been Falcon Crowdstrike and Carbon Black (VMWare). Others are capable of such activity. A partial list of popular Endpoint Protection solutions is:
- Falcon® by Crowdstrike
- Carbon Black® by VMWare
- Harmony® by Checkpoint
- Sophos Internet X®
- Symantec E.S. ®
- Trend Micro Apex One®
- Qualys
- Tenable Nessus®
- Fire Eye® by Trellix
Not all of these applications have been associated yet with application stoppage, but should be monitored if SAS suddenly stops working, with no other apparent changes (patching, software upgrades, etc.).
System Monitoring Applications
System Monitoring Applications actively monitor systems for workload analysis, resources used, and system outages. They generate threshold alerts when detecting failed or out-of-resource conditions. Most System Monitoring Applications have not been found to interfere with Application performance or execution. In evaluation by SAS IT, Dynatrace was found to interfere with SAS LASR activity. Recently, an unpatched Dynatrace version stopped SAS LASR from executing at a Large Bank, disabling Financial Compliance Reporting for five days.
We disabled all third-party security and monitoring applications, and discovered a certificate monitoring extension in Dynatrace was the culprit deny LASR execution. A hot fix was requested from Dynatrace which alleviated the issue.
Solution
If your SAS Customer experiences their SAS Foundation or SAS User Applications suddenly not working, quickly check to see if they have recently installed, reconfigured, or implemented any of the above types of Endpoint Security, or System Monitoring software. Given our recent spate of incidents, we have resolved them quickly by:
- Making an assay of third-party products running on the system during SAS execution
- Investigating if other non-SAS applications are affected as well
- Using nmon, collect-l, or top to indicate resources actively used by those products
- Temporarily Disabling the products, one-by-one to determine if they are preventing SAS and/or other applications from running or performing adequately
If you suspect interference with SAS operations from these applications, please ensure a Service Now Ticket is filed with Technical Support for investigation and resolution.
