High Availability and TLS with SAS Workload Orchestrator

2 Likes

With the introduction of SAS Workload Orchestrator in SAS 9.4 M6, customers have a new tool to manage the workload on their SAS Grid environment. However, as more and more customers are getting SAS Workload Orchestrator consultants in the field are being asked how to secure connections to it via TLS and how to make it highly available.

Out of the box, SAS Workload Orchestrator doesn’t use the SAS Web Server and isn’t installed on the SAS middle tier like the other SAS Web Applications. It is installed on the grid nodes and runs independently of the middle tier.

There are a couple of simple steps that you can take to ensure that SAS Workload Orchestrator is both secure and highly available.

Setup TLS to SAS Workload Orchestrator
Setup SAS Workload Orchestrator to be highly available
Setup TLS to SAS Workload Orchestrator via the SAS Web Server

Setting up TLS to SAS Workload Orchestrator

The first step to securing SAS Workload Orchestrator is to set up TLS. This will secure all external communication that comes in through the SWO web UI as well as all internal grid to grid communication. This process is pretty simple, but it does require the customer to meet a couple of pre-requisites:

The customer must provide a valid x509 certificate and private key with all grid nodes as subject alternative names (SANs) (although technically there are ways to avoid having all the grid nodes as SANs that leads to a much more complicated setup).
The customer must provide the root and intermediate certificates for constructing a CA chain file. These need to be imported via SAS Deployment Manager into the SAS truststore on all grid and mid-tier servers SAS installation directories (this is often shared on Unix grid nodes). In Windows these certificates must also be added to the Windows CA store: Add Your Certificates to the Windows CA Store.

Once you have the pre-requisites you are ready to begin the configuration. You'll first want to start with importing the CA Chain file into the SAS Truststore into each SAS-installation-directory for the grid nodes and middle-tier server via SAS Deployment Manager.

On Unix, you can place the certificate and private key in any permanent location. However, the following is the preferred location:

<SASHOME>/SASSecurityCertificateFramework/1.1/certificates

On Windows, the certificate with key must be added to the Windows Certificate store: Import the Client Certificate into the Windows Personal Machine Store

Modify <SASCONFIG>/Levn/Grid/sgmg_usermods.sh (UNIX) or sgmg_usermods.cmd (Windows) and define the environment variable USERMODS_OPTIONS, specifying the appropriate SSL options.

Unix:
- SSLCALISTLOC specifies the location of the trust store.
- SSLCERTLOC specifies a certificate that is valid for all of the grid nodes.
- SSLPVTKEYLOC specifies the certificate's key.
- SSLPVTKEYPASS specifies the password for the certificate's key. If the certificate's key is password protected, this option must be provided.
Windows:
- SSLCERTISS Specifies the name of the issuer of the digital certificate that TLS should use.
- SSLCERTSERIAL Specifies the serial number of the digital certificate that TLS should use.

UNIX:

USERMODS_OPTIONS='-SSLCALISTLOC "SASHome/SASSecurityCertificateFramework/1.1/cacert/trustedcerts.pem" -SSLCERTLOC "directory_path/certificate.crt" -SSLPVTKEYLOC "directory_path/certkey.key"

Windows:

set USERMODS_OPTIONS=-sslcertiss "Example Issuing Certificate CA" -sslcertserial 100001F00E0A0B0A0000D00DDF00000001E00F

Next, in the SAS Workload Orchestrator Web Interface under Configuration, uncheck the SSL disable switch...

and click the Save button

Then on Unix, stop the SAS Workload Orchestrator daemon on each node by running:

<SASConfig>/Levn/Grid/sgmg.sh stop

or using the gridStop.py script:

<SASConfig>/Levn/Grid/gridStop.py

On Windows, you can stop the service on each node or use the PowerShell script:

<SASConfig>\Levn\Grid\gridStop.ps1

Now we are done with modifications on the SAS Workload Orchestrator daemon's configuration, but we still have a couple of modifications to make in SAS Management Console to make other processes aware of the new configuration.

First, we want to modify the connection properties for the SASApp - Grid Server

Then modify the Grid Options to set noSSL=0.

Now it's time to update the connection property from HTTP to HTTPS. This is under SAS Management Console -> Application Management -> Configuration Manager -> right click Workload Orchestrator -> select properties -> select Internal Connection tab.

Now we are done setting up TLS to SAS Workload Orchestrator.

The last step is starting SAS Workload Orchestrator again by going to all of the grid nodes starting the service (Windows) or on Unix running:

<SASCONFIG>/Levn/Grid/sgmg.sh start

or if configured using the gridStart.py script:

<SASConfig>/Levn/Grid/gridStart.py

on Windows you can also use the PowerShell script:

<SASConfig>\Levn\Grid\gridStart.ps1

You can also find this process for enabling TLS (as well as disabling it) in the documentation: Configure HTTPS Manually for SAS Workload Orchestrator

Additional Steps to Ensure SAS Workload Orchestrator is Highly Available

To make SAS Workload Orchestrator highly available you will want to proxy it through the SAS Web Server. Rob Collum wrote a post where he discussed setting up SAS Workload Orchestrator for high availability. However, that architecture still resulted in a single point of failure (the SAS Web Server).

The architecture below alleviates that single point of failure by setting up high availability of the SAS Middle Tier. See the SAS Middle Tier Administration Guide for the process to establish high availability for the SAS 9 Web Server. Then implement your swo.conf file on both middle-tier SAS Web Servers. Finally, create an additional load balancer rule to direct the VIP to load balance between the SAS Web Servers. Just as the SAS Web Server was configured to listen on port 8901 to match SWO, do the same for the VIP provider as well - while any available port number is acceptable, the consistency will be easier to understand. Also, ensure that sticky sessions are enabled on the VIP provider, too.

Once you have all of that setup there is still one last piece you need to complete to access SAS Workload Orchestrator via the load balancer. This is simply modifying the connection properties in SAS Management Console as shown above (SAS Management Console -> Application Management -> Configuration Manager -> right-click Workload Orchestrator -> select Properties -> select Internal Connection tab) to modify the hostname to be the load balancer.

After these changes have been made instead of accessing SAS Workload Orchestrator through the SAS Web Server hostname: https://<SAS_Web_Server>:8901 you will access it via the load balancer: https://<Load_Balancer>:8901

Setting up TLS to SAS Workload Orchestrator to via the SAS Web Server

The sample swo.conf file below also has the necessary TLS options for when you have TLS setup to the SAS Web Server. Note that <SASHome>,host_candadateX, host_port and the ssl/ files would need to be set accordingly for the environment.

#IfModule !watchdog_module
<IfModule !watchdog_module>
LoadModule watchdog_module "<SASHome>/SASWebServer/9.4/httpd-2.4/modules/mod_watchdog.so"
</IfModule>

# Load the health check module.
<IfModule !proxy_hcheck_module>
LoadModule proxy_hcheck_module "<SASHome>/SASWebServer/9.4/httpd-2.4/modules/mod_proxy_hcheck.so"
</IfModule>

# LogLevel proxy_hcheck:TRACE8

# Specify a port for SAS Workload Orchestrator traffic.
Listen 8901 https

# Configure a virtual host (using the port specified above) to route traffic 
# to SAS Workload Orchstrator.
<VirtualHost *:8901 >

  # We use a health check to ensure that only the current master is available. All other hosts 
  # will fail the health check and be disabled. Only the current master will return 401 on 
  # GET /sasgrid/index.html.
  # All other hosts will return 301.
  ProxyHCExpr ok401 {%{REQUEST_STATUS} == 401 || %{REQUEST_STATUS} == 200}

  ProxyPass / balancer://SASWorkloadOrchestrator/
  ProxyPassReverse / balancer://SASWorkloadOrchestrator/

  SSLEngine on
  SSLProxyEngine on
  SSLProxyVerify require
  SSLProxyVerifyDepth 10
  SSLProxyCACertificateFile "ssl/ca_chain.pem"
  SSLCertificateFile "ssl/.crt"
  SSLCertificateKeyFile "ssl/.key"

  # The health checks ensures that all traffic will be routed to the current master. However, 
  # we can optimize even further by assigning each balancer member to a balancer member set (where 
  # each set contains only a single host). This ensures that Apache will traverse the list of 
  # master candidates in the same order as SAS Workload Orchstrator.

<Proxy balancer://SASWorkloadOrchestrator>
BalancerMember http://host_candidate1:host_port lbset=0 hcinterval=5 
hcmethod=GET hcuri=/login/login.html hcexpr=ok401
    BalancerMember http://host_candidate2:host_port lbset=1 hcinterval=5 
hcmethod=GET hcuri=/login/login.html hcexpr=ok401
    BalancerMember http://host_candidate3:host_port lbset=2 hcinterval=5 
hcmethod=GET hcuri=/login/login.html hcexpr=ok401

</Proxy>

</VirtualHost>

You can also find this process for enabling access to SAS Workload Orchestrator though the SAS Web Server documented here: Enabling Master Host Failover for Job Flow Scheduler

Conclusion

SAS Workload Orchestrator provides a new way for customers to manage their grid environment. However, with this new solution comes new ways to secure and set up a highly available environment. Both of which are very important to a lot of our SAS Grid customers. When architecting an environment for a customer it is important to understand these new tools and ensure that the environment is architected to meet the customer's requirements.

Thanks to Rob Collum and Edoardo Riva for helping me put this post together.