BookmarkSubscribeRSS Feed
MargaretC
SAS Employee

Reference Name: Security Software disabling SAS Applications
Severity: High
Status: New and Improved Third-Party Security and System Monitoring Applications are constantly being introduced and evolving. As they do, their configuration and activities can interfere with, or stop SAS applications from executing.  This is an ongoing problem and is tracked continuously.  Periodically review this note for changes or additions, and advice. 

 

Impact

SAS Foundation Applications like LASR, base SAS, SAS GRID,; and SAS End User Applications like VA, VI, VS, etc. can be hampered from performing well, or completely denied resources or permission to run by the configurations and activities of Third-Party Security or Monitoring Software.  This can completely disable the SAS Application, resulting in business stoppage. 

 

Description

A recently high number of SAS Users engaged SAS Technical Support to discover why their SAS applications/jobs were not running.  Most of the situations involved long-standing, existing systems, which had recently upgraded their OS, Security Software, and/or added System Monitoring.  The systems had added or made changes to Endpoint Security Protection or Monitoring Software:

 

Endpoint Security Protection Software

Endpoint security monitors and protects endpoints (workstations, servers, phones, tablets, networks, switches, etc.) that can be the entry point of malware, cyberattacks, and other security invasions.   It allows organizations to protect their computer networks, networks, and data for hackers and malware.  In securing these endpoints, the solution actively monitors, detects threats, investigates, and responds to unsafe or suspicious behavior.  This can include activities like virus checking and containment, malware interception, preventing applications from using resources, or stopping application executions that fall within pre-configured protocols.

 

Given the proliferation and adoption of Endpoint Security in todays fast moving security market, we have had 6 customers in the last two months have their applications disabled by Endpoint Protection software they installed.

 

The most frequent Endpoint Protection Applications encountered stopping SAS Applications from running have been Falcon Crowdstrike and Carbon Black (VMWare).  Others are capable of such activity.  A partial list of popular Endpoint Protection solutions is:

  • Falcon® by Crowdstrike
  • Carbon Black®  by VMWare
  • Harmony®  by Checkpoint
  • Sophos Internet X®
  • Symantec E.S. ®
  • Trend Micro Apex One®
  • Qualys
  • Tenable Nessus®
  • Fire Eye® by Trellix

 

Not all of these applications have been associated yet with application stoppage, but should be monitored if SAS suddenly stops working, with no other apparent changes (patching, software upgrades, etc.).

 

System Monitoring Applications

System Monitoring Applications actively monitor systems for workload analysis, resources used, and system outages.  They generate threshold alerts when detecting failed or out-of-resource conditions.  Most System Monitoring Applications have not been found to interfere with Application performance or execution.  In evaluation by SAS IT, Dynatrace was found to interfere with SAS LASR activity.  Recently, an unpatched Dynatrace version stopped SAS LASR from executing at a Large Bank, disabling Financial Compliance Reporting for five days. 

We disabled all third-party security and monitoring applications, and discovered a certificate monitoring extension in Dynatrace was the culprit deny LASR execution. A hot fix was requested from Dynatrace which alleviated the issue. 

 

Solution 

If your SAS Customer experiences their SAS Foundation or SAS User Applications suddenly not working, quickly check to see if they have recently installed, reconfigured, or implemented any of the above types of Endpoint Security, or System Monitoring software.  Given our recent spate of incidents, we have resolved them quickly by:

 

  • Making an assay of third-party products running on the system during SAS execution
  • Investigating if other non-SAS applications are affected as well
  • Using nmon, collect-l, or top to indicate resources actively used by those products
  • Temporarily Disabling the products, one-by-one to determine if they are preventing SAS and/or other applications from running or performing adequately

 

If you suspect interference with SAS operations from these applications, please ensure a Service Now Ticket is filed with Technical Support for investigation and resolution. 

1 REPLY 1
MargaretC
SAS Employee

The largest IT outage in history occurred on July 19, 2024, when an update by Falcon CrowdStrike® brought down the Microsoft Windows hosts it is supposed to protect.  This ran across many business industries including Banks, Airlines, Hospital Systems, and Retail Point-of-Sale Systems, among others.


Falcon CrowdStrike® is a third-party software company providing endpoint protection and anti-virus software.  Microsoft is one of its largest vendor partners/customers.  SAS, Red Hat, and other software companies have previously experienced disruptions from CrowdStrike denying their software components (including Open API calls) from executing.  Both SAS and Red Hat have been counseling our customers to re-configure CrowdStrike settings or disable it altogether on their systems to avoid execution stoppages. 

 

Please note SAS, cannot fix this issue, as it is the domain of the third-party Vendor, and the customer’s back-end IT Services Organizations.

 

Why only Microsoft Windows? The software push by CrowdStrike included an invalid memory reference to a null pointer, which causes Microsoft Windows OS to crash ungracefully.  It does not crash the Unix/Linux OS, and only results in a process error.  In addition, Microsoft entered into an agreement with the EU to make their Open API calls available to third-party software, and not just Windows Applications.  CrowdStrike and other endpoint protection, anti-virus software products were widely adopted after the EU passed laws for open and fair competition for Open API access.  In the last year we have seen numerous instances where CrowdStrike has affected SAS and Red Hat Systems.  

 

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 1 reply
  • 3069 views
  • 14 likes
  • 1 in conversation