Automated Patching of Log4j Vulnerabilities in SAS 9.4 and SAS Viya 3.x

3 Likes

You have undoubtedly heard about the Log4shell vulnerability and how important it is to patch susceptible systems. In response to the CVE alert, SAS has released a SAS Statement Regarding Remote Code Execution Vulnerability (CVE-2021-44228) which describes the issue, its impact on SAS software, and provides guidance and recommendations for remediation steps. For administrators, SAS has also published Instructions for the SAS Response to Log4j Vulnerabilities which includes detailed instructions for remediation steps.

While you can certainly locate and patch affected files in your deployment using tedious, repetitive manual steps, SAS recommends using loguccino for SAS 9.4 and SAS Viya 3.x deployments to automate the scan and patch process. Loguccino, which is a fork of logpresso/CVE-2021-44228-Scanner, has been specially modified to work with SAS software. Loguccino traverses all subdirectories in an identified path, including recursively traversing all nested .tar.gz, .tgz, .tar, .zip, .ear, .war and .jar archives, and reports or patches the affected files it finds. Using loguccino makes locating and patching vulnerable files quite easy which should help administrators keep their deployments protected from malicious activity.

In this article, we will use loguccino to scan and patch a SAS 9.4 deployment and look at how its capabilities can be scripted to patch a multi-machine deployment.

Process

The loguccino patching process is outlined in this diagram. You will want to patch your system today but also repeat the patching steps after each change to your deployment to make sure vulnerabilities are not re-introduced. It is very likely that subsequent scans will report no problems but it is better to be safe than take a chance that a later update re-introduces vulnerabilities into your deployment.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Prepare

There are a few easy steps to prepare your deployment for patching.

Go ahead and apply any outstanding hotfixes or updates to your system before you patch it. This extends the effectiveness of each patching.
Stop your deployment. Patching a file on disk does not patch the version of the program already running on the host so you should run loguccino against a stopped environment.
Perform a backup. As always, having a backup protects you in case there is a problem during the patch process.
Download loguccino. There is a Linux binary or a jar file if you have a Windows deployment. The two are identical in functionality.

Scan for Vulnerabilities

Once you have completed the preparation steps it is time to scan your deployment for files that are vulnerable to the Log4j issues. Scanning is a single command that requires the name of the path to search for affected files. In this case, I am going to pass in the SAS configuration directory so everything below this path will be examined. The --no-pretty option suppresses console animation that occurs during the scan.

C:\bin\loguccino>java -jar loguccino-all.jar scan --no-pretty c:\sas\config\lev1

INFO: Results written to CSV file: C:\bin\loguccino\loguccino-scan-06012022103210.csv

Windows deployments will require a second scan for the SASHOME directory.

e.g., C:\bin\loguccino>java -jar loguccino-all.jar scan --no-pretty "C:\Program Files\SAS"

On Linux, you need only one scan as /opt/sas is the parent directory for both the SASHome and the configuration directory.

e.g., /tmp/loguccino scan --no-pretty /opt/sas

Important: You will need to scan and patch all machines in a multi-machine deployment.

View results

As noted in the message above, results of the scan are provided in a CSV file which is subsequently used as input to the patch process for remediation. In the scan results of my SAS 9.4 system I see several files have been identified for patching. Notice the far right column informs me that none of these files have been patched (Patched=FALSE).

View log

It is a good idea to review the log of your scan to see if there were any files that could not be examined. In my case, there were two files that could not be read. I either need to assess whether they present a vulnerability and manually patch them if they do or modify permissions on the files to make them readable.

10:36:39.715 ERROR: [*] CVE-2021-44228 vulnerability in c:\sas\config\lev1\Web\Temp\LoadLocaledValuesTempJarLoc\log4j-core-2.1.jar, log4j 2.1.0
10:37:13.197 ERROR: [*] CVE-2021-44228 vulnerability in c:\sas\config\lev1\Web\WebAppServer\SASServer1_1\lib\log4j-core-2.1.jar, log4j 2.1.0
10:38:17.389 INFO: Scanned 19588 directories and 357716 files
10:38:17.410 INFO: Found 20 vulnerable files
10:38:17.410 INFO: Found 0 patched files
10:38:17.417 ERROR: Failed to read 2 files:
1 -> c:\sas\config\lev1\Web\SASEnvironmentManager\agent-5.8.0-EE\bundles\agent-5.8.0\product_connectors\rt-1.0.2.tar.gz
2 -> c:\sas\config\lev1\Web\SASEnvironmentManager\agent-5.8.0-EE\bundles\agent-5.8.0\product_connectors\snmp-1.0.2.tar.gz
10:38:17.418 INFO: Results written to CSV file: C:\bin\loguccino\loguccino-scan-06012022103210.csv

Patch the Vulnerable Files

Now that we have a list of vulnerable files, we simply need to feed that list back into the patch command and let loguccino remove the Log4j issues from these problematic files.

C:\bin\loguccino>java -jar loguccino-all.jar patch --no-pretty loguccino-scan-06012022103210.csv

INFO: Patch results written to CSV file: C:\bin\loguccino\loguccino-patch-06012022105042\patch.csv

View patch.csv

The patch results are stored in a time-stamped patch directory that contains backups of the original files as well as working files and a summary CSV file. The patch.csv file lists all of the affected files that were patched and identifies the backup location for each.

Rerun scan

If you want confirmation that your system has been patched you can re-scan your directories.

C:\bin\loguccino>java -jar loguccino-all.jar scan --no-pretty c:\sas\config\lev1

INFO: Results written to CSV file: C:\bin\loguccino\loguccino-scan-06012022111248.csv

View results

The results this time show that all of the files that could contain vulnerabilities have been patched (Patched=TRUE in the right-most column). As before, you should also review the scan log file to see if there were any files that could not be read.

Scan and patch SASHome

As I have a SAS 9.4 deployment on Windows, I would now want to repeat the process by scanning and patching my SASHome directory. Once that is complete I can restart my deployment.

Updating Multi-Machine Deployments

As mentioned earlier, you will need to scan and patch each machine in a multi-machine deployment. Loguccino is easy to script and has an --output parameter that allows you to explicitly name the resulting CSV files, making it easy to know which file to pass into the patch command. As an example, my colleague Stuart Rogers put together log-remediation.sh, a script that can be used to automate the scan and patch process.

#!/bin/bash
# Script to run against Viya 3.5 and 9.4 to remediate Log4J issues

# Fetch argument as SAS directory
if [ -z $1 ]; then
echo "You must specify the SAS directory to scan & patch"
exit 0
else
mySASDIR=${1}
fi

if [ ! -d "$mySASDIR" ]; then
echo "The SAS directory: ${mySASDIR} does not exist!"
exit 0
fi

# Fetch latest tool
myDIR="/tmp/log4j"
myURL="https://github.com/sassoftware/loguccino/releases/latest/download/loguccino"

if [ -d "$myDIR" ]; then
rm -rf ${myDIR}
fi
mkdir -p ${myDIR}
curl -skL --output ${myDIR}/loguccino ${myURL}
chmod a+x ${myDIR}/loguccino

# Run tool
# Start with scan
pushd ${myDIR}
${myDIR}/loguccino scan --no-pretty --output ${myDIR}/loguccino.csv ${mySASDIR}

# Patch
umasktmp=$(umask)
umask 022
pushd ${myDIR}
${myDIR}/loguccino patch --no-pretty --no-compress ${myDIR}/loguccino.csv
umask ${umasktmp}

Once the script is on each machine, I can use Ansible or something similar to run log4j-remediation.sh on each machine using a command such as this one.

/usr/bin/ansible -i /root/sas_viya_playbook/inventory.ini all -m script -a "/opt/scripts/log4j-remediation.sh /opt/sas" --become --become-user sas

I hope this helps you keep your SAS deployments safe and protected.

Resources

Special thanks to @StuartRogers for his example scripts.

Find more articles from SAS Global Enablement and Learning here.

SASKiwi · ‎01-20-2022

@ScottMcCauley - Excellent writeup. Nice to have independent validation of the approach we have been taking. One problem we ran into was not having a loguccino-compatible version of Java on older SAS installations. In that case we simply copied the contents of the SAS install folder SASPrivateJavaRuntimeEnvironment\9.4 from a server with a compatible version to the server with the non-compatible version into new folder say SASPrivateJavaRuntimeEnvironment\9.4_loguccino. Then you just run the Java executable in 9.4_loguccino rather than 9.4.

ScottMcCauley · ‎01-21-2022

Excellent point, thanks for sharing your solution!

Denise · ‎02-16-2022

Our software packaging team (they package into automation the client software distribution process) keeps asking me if we will get a new depot with an upgraded Deployment Manager (2.17). Does anyone have information about an updated deployment manager?

SASKiwi · ‎02-16-2022

@Denise - If you want an updated depot containing a new version deployment manager then typically you would request that from your SAS account manager.