Shaping Network Traffic for CAS in SAS Viya 3.5
SAS Viya 3.5 introduced a new approach for defining network configuration for SAS host machines. The new approach improves the consistency of how networking is applied which in turn leads to substantial benefits including more flexibility for different network topologies and better resiliency in the face of network changes. See my previous post, Understanding Network Binding Variables in Viya 3.5, for more details.
The idea is that your Viya host(s) will often have more than one network interface card (NIC). Depending on how your customer IT has configured their use, those cards could be used to separate traffic into and out of the machine, essentially providing a dedicated i/o path for traffic that's high priority, requires low latency, needs to be isolated from other networks, or is of large volume to prevent it from competing with general communication of other activity.
As it turns out, SAS Cloud Analytic Services (CAS) often fits the profile of network usage for all of those reasons. Let's look at configuring CAS to optimize its communication with other services and its movement of data.
Directing CAS with new configuration
By default, CAS relies on the same set of network configuration variables defined by the new network.conf technique (i.e. SAS_BIND_ADDR, SAS_EXTERNAL_BIND_ADDR, and related). Often these are sufficient for nominal usage. But sometimes, it might be necessary to split out data traffic because it can be so voluminous that it saturates the network, preventing timely response to other requests and activity. CAS offers some additional parameters to make it happen.
There are a few files to be familiar with when making changes to the configuration.
vars.yml
The vars.yml file is provided in the SAS Viya Playbook which is placed on the Ansible Controller host machine. It's referenced when running the site.yml file which performs the deployment of Viya software. The parameters discussed in this post can all be defined in the CAS_CONFIGURATION section of vars.yml.
It is considered a proven practice to keep vars.yml up-to-date with configuration changes after deployment as well.
Changes made to vars.yml require re-running the site.yml deployment for them to go into effect - which means a full Viya outage while it runs.
There are times when you might need to make targeted configuration changes. And so there are two more files to consider to help with that:
casconfig_usermods.lua
This file exists on the CAS controller host and is read by the controller at start up. The values it finds here override the standard configuration files and are then shared identically with all workers.
So if you have a universal parameter that works for all CAS nodes, then place it here.
Changes made to this file require a restart of CAS to go into effect. The rest of Viya is unaffected.
node_usermods.lua
Each CAS node has an instance of this file that it reads at start up. These values override anything provided by the CAS controller (as specified in casconfig_usermods.lua).
So if you have a parameter with a value unique to this CAS host, then place it here.
Changes made to this file require a restart of CAS to go into effect. The rest of Viya is unaffected.
Selecting network identity
At deployment, CAS is configured by the Viya 3.5 network configuration. The hostnames and IP addresses it references are originally driven by the SAS_HOSTNAME, SAS_BIND_ADDR and other parameters as well as their _EXTERNAL_ equivalents for public interfaces. So, for example, the cas.hosts file should populate with hostnames derived from the Viya 3.5 network config.
There are situations when the hostnames and/or IP addresses that CAS uses to reference its own nodes are not compatible with external data providers which might reside on a different subnet (or otherwise find CAS' hostnames and IP addresses unroutable). So CAS provides configuration parameters to override those default values.
cas.dcHostnameResolution="<ep | ep_fqdn | cas | cas_ipv6>"
This parameter directs which software service will resolve hostnames into IP addresses for the SAS Embedded Process to use.
Default is "cas". This directs the CAS controller to resolve its nodes' names into IPv4 addresses and send the list over to the EP when requesting data. The EP will attempt to connect to those IP addresses directly. The "cas_ipv6" value works the same, but the CAS controller sends over IPv6 addresses for its nodes instead.
However, if the EP is in a subnet where those IP addresses aren't routable to CAS, then we can direct CAS to send over hostnames as listed in its cas.hosts file for the workers with cas.dcHostnameResolution="ep". Or specify "ep_fqdn" which directs CAS to send its workers' fully-qualified domain names over to the EP.
By using one of the the "ep" values, it might mean that the names the EP resolves point to a different set of IP addresses than CAS knows about. Same name, but different IP addresses is somewhat unusual and assumes separate DNS, virtualized networking, local hostname resolution files, or some other extra bit of mapping outside of Viya.
To be clear, it's certainly possible that the hostname/FQDN resolution on the EP would yield the exact same set of IP addresses that CAS would provide when cas.dcHostnameResolution="cas". And if so, that's fine, too.
cas.extHostKnownBy="<cas-node-id.site.com>"
This parameter provides an additional FQDN for the CAS host. If needed, be sure to specify this parameter in node_usermods.lua for each CAS host.
Default is unspecified. This means that CAS relies on the values from its cas.hosts file to advertise its hosts names, FQDN, and/or IP addresses.
Use this parameter when you want to provide a substitute name for the CAS controller to provide for this host, instead of how it's listed in cas.hosts file, when it sends the list of hosts over to the external data provider. This allows the external data provider to resolve the hostname/FQDN to an IP address using its own local lookup service (DNS, /etc/hosts file, or similar).
Selecting network ports
Within a single machine, network services each listen at their own port. Some services, like CAS, can listen on several ports simultaneously for different tasks. Depending on what communication needs further tweaking, CAS offers a selection of port configurations to help ensure traffic is routed the best way. The parameters shown here can be defined at initial install in the CAS_CONFIGURATION section of vars.yml or after install in either casconfig_usermods.lua or node_usermods.lua files, depending on specificity.
cas.port="<0 - 65535>"
Default is 5570 (TLS encrypted). 0 requests an ephemeral port assignment from the OS.
This is the port that the CAS controller listens to for client requests. No CAS workers use this value. Avoid using a value of zero for this parameter as this port number is one that you need to know in order to advertise it to non-Viya clients.
cas.httpport="<0 - 65535>"
Default is 8777 (TLS encrypted). 0 requests an ephemeral port assignment from the OS.
Similar to cas.port, this is another port that the CAS controller listens to for client requests over HTTP. No CAS workers use this value. Avoid using a value of zero for this parameter as this port number is one that you need to know in order to advertise it to non-Viya clients.
cas.gcPort="<0 - 65535>"
Default is 0. 0 requests an ephemeral port assignment from the OS with the resulting assignment in the range of 32678–61000. If set explicitly, the typical value used is 5580.
The gcPort is CAS' network channel for general communications between the CAS hosts. It's what the controller(s) and workers use to talk to each other.
cas.dcPort="<-1,0 - 65535>"
Default is -1 which means use the same value as cas.gcPort. 0 requests an ephemeral port assignment from the OS.
This is the data communication port. Specifying a separate port for data transfer i/o from external data providers can improve performance by reducing contention for a single port (i.e. the gcPort).
cas.dcExtPort="<0 - 65535>"
Default is 0. Unlike the other port parameters, zero doesn't request an ephemeral port, instead it directs CAS to specify the same value as dcPort (similar to how dcPort="-1" directs CAS to use the gcPort for data traffic).
Note that this directive doesn't define a new port for CAS to listen on. This is the port number that CAS will inform an external data provider, like the EP, to use when it initiates data transfer communication to the CAS workers. This is useful for situations when the dcPort isn't accessible from the outside - like when CAS is running in a containerized environment. The dcExtPort value will direct CAS to advertise the mapped port instead. That's why there's no ephemeral option. For non-containerized environments, expect to leave this with default value.
Summarizing CAS port parameters:
| Parameter | Purpose | Applies to | Default value | Value's meaning | Encryption |
| cas.port | Client communication | Controller only | 5570 | Yes by default | |
| cas.httpPort | Client communication | Controller only | 8777 | Yes by default | |
| cas.gcPort | General intra-node communication | Any CAS node | 0 | Use ephemeral port | Optional |
| cas.dcPort | Data connector communication | Any CAS node | -1 | Use gcPort | Optional |
| cas.dcExtPort | Data connector communication | Any CAS node | 0 | Advertise dcPort's value to external data providers | Optional |
Scenarios
Okay, I've explained the documentation and theory a bit. Let's look at some common scenarios and how to deal with them.
Default
Without specifying any new values to the parameters described above, then you can expect an MPP deployment of CAS will behave as:
- CAS controller listens for TLS encrypted, native client traffic on TCP port 5570.
- CAS controller listens for TLS encrypted, HTTP RESTful traffic on TCP port 8777.
- All CAS intra-node traffic is not encrypted - between controller(s) and workers - on ephemeral ports (cas.gcPort) designated when CAS processes start up.
As a point of interest, CAS intra-node traffic includes:
- Controller directing workers on tasks
- Controller distributing data to workers from a data load
- Workers sharing data between them (rare)
- Workers returning their interim results to the controller
- And so on.
Encryption of CAS intra-node traffic (i.e. cas.gcPort) can be enabled - be sure to allow for an accompanying slowdown of throughput performance - with the env.CAS_Internode_Data_SSL parameter.
Significant data transfer to/from data sources
Change cas.dcPort in casconfig_usermods.lua and/or node_usermods.lua.
CAS is a powerful, multi-threaded runtime engine which can support many concurrent users, sessions, and operations. Data movement of large volumes can consume considerable network resources and potentially slow other communications traffic between the CAS hosts.
For an environment where data transfer from external sources is a substantial (or high latency) workload, consider setting the cas.dcPort value to a value other than -1 to split the communication from outside data providers (through the SAS data connectors) to a different port interface.
Encryption of cas.dcPort traffic can be enabled if needed using available caslib options.
Separate data transfer from data sources to a different I/O channel
Change cas.dcExtHostname in node_usermods.lua.
Assuming your CAS host(s) has more than one NIC, then we can configure CAS to direct an external data provider (like the EP) to send data over a different network interface than CAS uses for its normal, intra-node communication. This further separates traffic onto a different physical path for improved throughput performance.
When CAS for Viya 3.5 is deployed, the cas.hosts file is built using values derived from SAS_HOSTNAME and SAS_BIND_ADDR in the new network configuration. And those specify the normal route (that is, which NIC card) that CAS will use for its general communications.
For CAS hosts with more than one NIC, consider setting cas.dcExtHostname to a value which resolves to specify the CAS host's alternate NIC. Take care with this approach to ensure that the alternate NIC is intended for this use (not an admin-only interface, on the correct subnet, speed-rated sufficiently high to feed CAS with large data volumes, etc.)
SAS Embedded Process on a different subnet
For SAS Data Connector Accelerators, change cas.dcHostnameResolution in casconfig_usermods.lua.
It's fairly common for the SAS In-Database Embedded Process to reside in a data provider (Hadoop, Teradata) placed on a subnet which isn't directly routable to the IP addresses used by CAS.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
When a load action of data from the EP is executed, the default behavior of CAS is to send the EP a list of the workers' IP addresses. Then each instance of the EP distributes its allotment of data across all of the CAS workers at those IP addresses. If those IP addresses aren't routable, then modify the cas.dcHostnameResolution parameter to "ep" or "ep_fqdn". This causes CAS to send a list of hostnames (or their equivalent FQDN) as referenced by the cas.hosts file over to the EP instead of IP addresses. Then the EP itself can resolve them to IP addresses which would routable from that end.
This approach necessitates some additional OS or DNS network configuration by your customer's IT team, similar in concept to dedicated /etc/hosts files (or accomplished in DNS or other way).
Encryption of traffic between CAS and the EP can be enabled with the cas.dcTCPMencrypt parameter on both CAS and EP hosts.
CAS running in containers
Viya 2020 is due to be released a little later this year. And as we all know, it'll employ a wholly new approach to deployment, running inside Docker containers and orchestrated by Kubernetes. In Viya 3.5, it is also possible to employ a containerized approach to running CAS.
One (of many) key things to understand about containers is that, by default, they are completely isolated from the outside world, even their host OS. So when you reference things like network ports inside a container, they only exist there, and aren't directly accessible to the outside world. Of course, that's not very useful in most cases, so you can define mappings to resources likes network ports to make them accessible.
As we established earlier, the CAS controller typically listens for HTTP-based requests on port 8777. If we place CAS inside a container, then we must define a port mapping to allow requests from outside the container to reach that port. The simplest approach is to define a mapping where Docker listens on the host OS port 8777 and forwards those to CAS listening on 8777 inside the container.
But what if there's something else in the host OS already using port 8777? Well, we can define Docker to listen at some other port number, like 7888 (or whatever) and anything sent there will forward to CAS' port 8777 inside its container. In that case, CAS does not know about port 7888 - as that's outside its containerized world.
For data from external providers:
- Specify static port values for cas.gcPort and cas.dcPort
- If needed, specify cas.dcExtPort and cas.dcExtHostname in node_usermods.lua.
Now change gears to consider cas.gcPort (default is ephemeral) and cas.dcPort (default is to use gcPort). So when CAS informs external data providers about the dcPort it's listening on for data connections, it'll advertise the current port number inside its container, but that will only work if a port mapping exists where Docker is listening for that traffic on the exact same port number in the host OS.
In containerized environment, specify a static (not ephemeral) value for cas.gcPort and cas.dcPort to use for the Docker port mappings. And then, if needed, specify cas.dcExtPort to tell CAS what that external port mapping value is that it should advertise to an external data provider when sending data.
A similar concept applies to CAS' hostname values inside the container as well. If they differ from what an external data provider would use to route traffic, then specify cas.dcExtHostname on each CAS node to inform CAS of the external hostname (or IP address) that should be advertised outside of the containerized environment.
Remember also that cas.dcHostnameResolution="ep" causes CAS to send over hostnames based on the contents of the cas.hosts file for the EP to resolve into IP addresses. When cas.dcExtHostname is provided for a CAS node, then that value will override the entry from the cas.hosts file which gets sent to the EP.
Coda
As you can see, the combination of CAS roles, various ports to listen for different kinds of traffic, deployment topologies, and virtualization technologies requires numerous configuration parameters to ensure everything gets to where it needs to go. Furthermore, the technique CAS relies on when transferring data from the SAS In-Database Embedded Process differs significantly from most typical client-server interactions... and this necessitates additional considerations as well.
For more information about the related topics in this article, refer to:
- Raphaël Poumarede's post, Working with multi-homed CAS clusters and the Embedded Process
- Erwan Granger's many GEL posts about SAS Viya running in containers
SAS documentation:
- SAS® 9.4 and SAS® Viya® 3.5 Programming Documentation > setServOpt Action
- SAS® Viya® 3.5 Administration: SAS® Cloud Analytic Services > Configuration File Options
- Encryption in SAS® Viya® 3.5: Data in Motion > Configure CAS Internode TLS
- SAS Support Site > SAS® for Containers Focus Area
