BookmarkSubscribeRSS Feed

Shaping Network Traffic for CAS in SAS Viya 3.5

Started ‎05-18-2020 by
Modified ‎07-03-2020 by
Views 3,902

SAS Viya 3.5 introduced a new approach for defining network configuration for SAS host machines. The new approach improves the consistency of how networking is applied which in turn leads to substantial benefits including more flexibility for different network topologies and better resiliency in the face of network changes. See my previous post, Understanding Network Binding Variables in Viya 3.5, for more details.

 

The idea is that your Viya host(s) will often have more than one network interface card (NIC). Depending on how your customer IT has configured their use, those cards could be used to separate traffic into and out of the machine, essentially providing a dedicated i/o path for traffic that's high priority, requires low latency, needs to be isolated from other networks, or is of large volume to prevent it from competing with general communication of other activity.

 

As it turns out, SAS Cloud Analytic Services (CAS) often fits the profile of network usage for all of those reasons. Let's look at configuring CAS to optimize its communication with other services and its movement of data.

Directing CAS with new configuration

By default, CAS relies on the same set of network configuration variables defined by the new network.conf technique (i.e. SAS_BIND_ADDR, SAS_EXTERNAL_BIND_ADDR, and related). Often these are sufficient for nominal usage. But sometimes, it might be necessary to split out data traffic because it can be so voluminous that it saturates the network, preventing timely response to other requests and activity. CAS offers some additional parameters to make it happen.

 

There are a few files to be familiar with when making changes to the configuration.

vars.yml

The vars.yml file is provided in the SAS Viya Playbook which is placed on the Ansible Controller host machine. It's referenced when running the site.yml file which performs the deployment of Viya software. The parameters discussed in this post can all be defined in the CAS_CONFIGURATION section of vars.yml.

 

It is considered a proven practice to keep vars.yml up-to-date with configuration changes after deployment as well.

 

Changes made to vars.yml require re-running the site.yml deployment for them to go into effect - which means a full Viya outage while it runs.

 

There are times when you might need to make targeted configuration changes. And so there are two more files to consider to help with that:

casconfig_usermods.lua

This file exists on the CAS controller host and is read by the controller at start up. The values it finds here override the standard configuration files and are then shared identically with all workers.

 

So if you have a universal parameter that works for all CAS nodes, then place it here.

 

Changes made to this file require a restart of CAS to go into effect. The rest of Viya is unaffected.

node_usermods.lua

Each CAS node has an instance of this file that it reads at start up. These values override anything provided by the CAS controller (as specified in casconfig_usermods.lua).

 

So if you have a parameter with a value unique to this CAS host, then place it here.

 

Changes made to this file require a restart of CAS to go into effect. The rest of Viya is unaffected.

Selecting network identity

At deployment, CAS is configured by the Viya 3.5 network configuration. The hostnames and IP addresses it references are originally driven by the SAS_HOSTNAME, SAS_BIND_ADDR and other parameters as well as their _EXTERNAL_ equivalents for public interfaces. So, for example, the cas.hosts file should populate with hostnames derived from the Viya 3.5 network config.

 

There are situations when the hostnames and/or IP addresses that CAS uses to reference its own nodes are not compatible with external data providers which might reside on a different subnet (or otherwise find CAS' hostnames and IP addresses unroutable). So CAS provides configuration parameters to override those default values.

cas.dcHostnameResolution="<ep | ep_fqdn | cas | cas_ipv6>"

This parameter directs which software service will resolve hostnames into IP addresses for the SAS Embedded Process to use.

 

Default is "cas". This directs the CAS controller to resolve its nodes' names into IPv4 addresses and send the list over to the EP when requesting data. The EP will attempt to connect to those IP addresses directly. The "cas_ipv6" value works the same, but the CAS controller sends over IPv6 addresses for its nodes instead.

 

However, if the EP is in a subnet where those IP addresses aren't routable to CAS, then we can direct CAS to send over hostnames as listed in its cas.hosts file for the workers with cas.dcHostnameResolution="ep". Or specify "ep_fqdn" which directs CAS to send its workers' fully-qualified domain names over to the EP.

 

By using one of the the "ep" values, it might mean that the names the EP resolves point to a different set of IP addresses than CAS knows about. Same name, but different IP addresses is somewhat unusual and assumes separate DNS, virtualized networking, local hostname resolution files, or some other extra bit of mapping outside of Viya.

 

To be clear, it's certainly possible that the hostname/FQDN resolution on the EP would yield the exact same set of IP addresses that CAS would provide when cas.dcHostnameResolution="cas". And if so, that's fine, too.

cas.extHostKnownBy="<cas-node-id.site.com>"

This parameter provides an additional FQDN for the CAS host. If needed, be sure to specify this parameter in node_usermods.lua for each CAS host.

 

Default is unspecified. This means that CAS relies on the values from its cas.hosts file to advertise its hosts names, FQDN, and/or IP addresses.

 

Use this parameter when you want to provide a substitute name for the CAS controller to provide for this host, instead of how it's listed in cas.hosts file, when it sends the list of hosts over to the external data provider. This allows the external data provider to resolve the hostname/FQDN to an IP address using its own local lookup service (DNS, /etc/hosts file, or similar).

Selecting network ports

Within a single machine, network services each listen at their own port. Some services, like CAS, can listen on several ports simultaneously for different tasks. Depending on what communication needs further tweaking, CAS offers a selection of port configurations to help ensure traffic is routed the best way. The parameters shown here can be defined at initial install in the CAS_CONFIGURATION section of vars.yml or after install in either casconfig_usermods.lua or node_usermods.lua files, depending on specificity.

cas.port="<0 - 65535>"

Default is 5570 (TLS encrypted). 0 requests an ephemeral port assignment from the OS.

 

This is the port that the CAS controller listens to for client requests. No CAS workers use this value. Avoid using a value of zero for this parameter as this port number is one that you need to know in order to advertise it to non-Viya clients.

cas.httpport="<0 - 65535>"

Default is 8777 (TLS encrypted). 0 requests an ephemeral port assignment from the OS.

 

Similar to cas.port, this is another port that the CAS controller listens to for client requests over HTTP. No CAS workers use this value. Avoid using a value of zero for this parameter as this port number is one that you need to know in order to advertise it to non-Viya clients.

cas.gcPort="<0 - 65535>"

Default is 0. 0 requests an ephemeral port assignment from the OS with the resulting assignment in the range of 32678–61000. If set explicitly, the typical value used is 5580.

 

The gcPort is CAS' network channel for general communications between the CAS hosts. It's what the controller(s) and workers use to talk to each other.

cas.dcPort="<-1,0 - 65535>"

Default is -1 which means use the same value as cas.gcPort. 0 requests an ephemeral port assignment from the OS.

 

This is the data communication port. Specifying a separate port for data transfer i/o from external data providers can improve performance by reducing contention for a single port (i.e. the gcPort).

cas.dcExtPort="<0 - 65535>"

Default is 0. Unlike the other port parameters, zero doesn't request an ephemeral port, instead it directs CAS to specify the same value as dcPort (similar to how dcPort="-1" directs CAS to use the gcPort for data traffic).

 

Note that this directive doesn't define a new port for CAS to listen on. This is the port number that CAS will inform an external data provider, like the EP, to use when it initiates data transfer communication to the CAS workers. This is useful for situations when the dcPort isn't accessible from the outside - like when CAS is running in a containerized environment. The dcExtPort value will direct CAS to advertise the mapped port instead. That's why there's no ephemeral option. For non-containerized environments, expect to leave this with default value.

 

Summarizing CAS port parameters:

Parameter Purpose Applies to Default value Value's meaning Encryption
cas.port Client communication Controller only 5570   Yes by default
cas.httpPort Client communication Controller only 8777   Yes by default
cas.gcPort General intra-node communication Any CAS node 0 Use ephemeral port Optional
cas.dcPort Data connector communication Any CAS node -1 Use gcPort Optional
cas.dcExtPort Data connector communication Any CAS node 0 Advertise dcPort's value to external data providers Optional

Scenarios

Okay, I've explained the documentation and theory a bit. Let's look at some common scenarios and how to deal with them.

Default

Without specifying any new values to the parameters described above, then you can expect an MPP deployment of CAS will behave as:

  • CAS controller listens for TLS encrypted, native client traffic on TCP port 5570.
  • CAS controller listens for TLS encrypted, HTTP RESTful traffic on TCP port 8777.
  • All CAS intra-node traffic is not encrypted - between controller(s) and workers - on ephemeral ports (cas.gcPort) designated when CAS processes start up.

As a point of interest, CAS intra-node traffic includes:

  • Controller directing workers on tasks
  • Controller distributing data to workers from a data load
  • Workers sharing data between them (rare)
  • Workers returning their interim results to the controller
  • And so on.

Encryption of CAS intra-node traffic (i.e. cas.gcPort) can be enabled - be sure to allow for an accompanying slowdown of throughput performance - with the env.CAS_Internode_Data_SSL parameter.

Significant data transfer to/from data sources

Change cas.dcPort in casconfig_usermods.lua and/or node_usermods.lua.

 

CAS is a powerful, multi-threaded runtime engine which can support many concurrent users, sessions, and operations. Data movement of large volumes can consume considerable network resources and potentially slow other communications traffic between the CAS hosts.

 

For an environment where data transfer from external sources is a substantial (or high latency) workload, consider setting the cas.dcPort value to a value other than -1 to split the communication from outside data providers (through the SAS data connectors) to a different port interface.

 

Encryption of cas.dcPort traffic can be enabled if needed using available caslib options.

Separate data transfer from data sources to a different I/O channel

Change cas.dcExtHostname in node_usermods.lua.

 

Assuming your CAS host(s) has more than one NIC, then we can configure CAS to direct an external data provider (like the EP) to send data over a different network interface than CAS uses for its normal, intra-node communication. This further separates traffic onto a different physical path for improved throughput performance.

 

When CAS for Viya 3.5 is deployed, the cas.hosts file is built using values derived from SAS_HOSTNAME and SAS_BIND_ADDR in the new network configuration. And those specify the normal route (that is, which NIC card) that CAS will use for its general communications.

 

For CAS hosts with more than one NIC, consider setting cas.dcExtHostname to a value which resolves to specify the CAS host's alternate NIC. Take care with this approach to ensure that the alternate NIC is intended for this use (not an admin-only interface, on the correct subnet, speed-rated sufficiently high to feed CAS with large data volumes, etc.)

SAS Embedded Process on a different subnet

For SAS Data Connector Accelerators, change cas.dcHostnameResolution in casconfig_usermods.lua.

 

It's fairly common for the SAS In-Database Embedded Process to reside in a data provider (Hadoop, Teradata) placed on a subnet which isn't directly routable to the IP addresses used by CAS.

 

4-dcHR.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

 

When a load action of data from the EP is executed, the default behavior of CAS is to send the EP a list of the workers' IP addresses. Then each instance of the EP distributes its allotment of data across all of the CAS workers at those IP addresses. If those IP addresses aren't routable, then modify the cas.dcHostnameResolution parameter to "ep" or "ep_fqdn". This causes CAS to send a list of hostnames (or their equivalent FQDN) as referenced by the cas.hosts file over to the EP instead of IP addresses. Then the EP itself can resolve them to IP addresses which would routable from that end.

 

This approach necessitates some additional OS or DNS network configuration by your customer's IT team, similar in concept to dedicated /etc/hosts files (or accomplished in DNS or other way).

 

5-dcHR-DNS.png

 

Encryption of traffic between CAS and the EP can be enabled with the cas.dcTCPMencrypt parameter on both CAS and EP hosts.

CAS running in containers

Viya 2020 is due to be released a little later this year. And as we all know, it'll employ a wholly new approach to deployment, running inside Docker containers and orchestrated by Kubernetes. In Viya 3.5, it is also possible to employ a containerized approach to running CAS.

 

One (of many) key things to understand about containers is that, by default, they are completely isolated from the outside world, even their host OS. So when you reference things like network ports inside a container, they only exist there, and aren't directly accessible to the outside world. Of course, that's not very useful in most cases, so you can define mappings to resources likes network ports to make them accessible.

 

As we established earlier, the CAS controller typically listens for HTTP-based requests on port 8777. If we place CAS inside a container, then we must define a port mapping to allow requests from outside the container to reach that port. The simplest approach is to define a mapping where Docker listens on the host OS port 8777 and forwards those to CAS listening on 8777 inside the container.

 

1-same.png

 

But what if there's something else in the host OS already using port 8777? Well, we can define Docker to listen at some other port number, like 7888 (or whatever) and anything sent there will forward to CAS' port 8777 inside its container. In that case, CAS does not know about port 7888 - as that's outside its containerized world.

 

2-different.png

 

For data from external providers:

  • Specify static port values for cas.gcPort and cas.dcPort
  • If needed, specify cas.dcExtPort and cas.dcExtHostname in node_usermods.lua.

Now change gears to consider cas.gcPort (default is ephemeral) and cas.dcPort (default is to use gcPort). So when CAS informs external data providers about the dcPort it's listening on for data connections, it'll advertise the current port number inside its container, but that will only work if a port mapping exists where Docker is listening for that traffic on the exact same port number in the host OS.

 

In containerized environment, specify a static (not ephemeral) value for cas.gcPort and cas.dcPort to use for the Docker port mappings. And then, if needed, specify cas.dcExtPort to tell CAS what that external port mapping value is that it should advertise to an external data provider when sending data.

 

A similar concept applies to CAS' hostname values inside the container as well. If they differ from what an external data provider would use to route traffic, then specify cas.dcExtHostname on each CAS node to inform CAS of the external hostname (or IP address) that should be advertised outside of the containerized environment.

 

3-dcExtPort.png

 

Remember also that cas.dcHostnameResolution="ep" causes CAS to send over hostnames based on the contents of the cas.hosts file for the EP to resolve into IP addresses. When cas.dcExtHostname is provided for a CAS node, then that value will override the entry from the cas.hosts file which gets sent to the EP.

Coda

As you can see, the combination of CAS roles, various ports to listen for different kinds of traffic, deployment topologies, and virtualization technologies requires numerous configuration parameters to ensure everything gets to where it needs to go. Furthermore, the technique CAS relies on when transferring data from the SAS In-Database Embedded Process differs significantly from most typical client-server interactions... and this necessitates additional considerations as well.

 

For more information about the related topics in this article, refer to:

SAS documentation:

 

Comments

Very usefull! Thank you

Version history
Last update:
‎07-03-2020 12:04 PM
Updated by:
Contributors

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started