SAS Viya Cloud Data Exchange Deployment and Configuration (Part-2)
This is the second part of the series “SAS Viya Cloud Data Exchange deployment and configuration”.
The SAS Viya Cloud Data Exchange software is available with most of the SAS Viya bundle license. There is no separate license for Cloud Data Exchange components like Co-located Data Agent and Remote Data Agent services.
SAS Viya Cloud Data exchange have two types of Data Agents.
- Co-located Data Agent - Reside at SAS Viya platform environment.
- Remote Data Agent - Hosted at on-premises data center.
In last post, I discuss about Co-located Data Agent deployment and configuration. In this post I discuss Remote Data Agent deployment and configuration at on-premises server.
Deployment and configuration of Remote Data Agent Service
The Remote Data Agent is a containerized software and runs with docker runtime. The Remote Data Agent software and components are delivered by using container-manager. It’s a SAS provided tool available at my.sas.com and can be download using standard user profile. You need to download container-manager, Viya Asset ( zip file ), and Viya License file from my.sas.com to deploy Remote Data Agent at on-premises server.
You can deploy more than one Remote Data Agent on on-premises server with same Viya license. More than one Remote Data Agent can be associated with one single Viya platform.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Pre-requisite for on-premises server to host Remote Data Agent
- Co-located Data Agent at Viya platform configured with data agent credential.
- Docker Version 18.09.6 or later
- Container Manager utility tool (Zip file)
- SAS Viya License and Asset file (Zip file)
- SSL Certificate for Remote Data Agent Server
- Public Ip for On-Premises Remote Data Agent Server
- On-Premises Firewall open for port used by the Remote Data agent ( Default #25141)
Steps to deploy and configure the Remote Data Agent Server.
Generate SSL certificate for Remote Data Agent Server
The Remote Data Agent communicate with Viya platform over a secured and encrypted network connection. To facilitate the secured and compatible connection the Remote Data Agent service SSL certificate must be signed by the same root cert authority who have signed for Viya platform. You can use self-signed certificate to start the Remote Data Agent service, but it’s recommended to customer provided Certificate. The Certificate could be generated using a third-party vendor tool or using open SSL tool.
Following are the steps to create a customized SSL Certificate for Remote Data Agent.
Create a Certificate Signing Request.
REMOTE_SERVER=”MyServer.test.com”
openssl req -newkey rsa:2048 -keyout ~/certs/rda.key -out ~/certs/rda.csr \
-subj "/C=US/ST=North Carolina/L=Cary/O=SAS/CN=$REMOTE_SERVER" \
-passout pass:mypass
Sign the CSR using the root CA private key and certificate.
While signing the certificate, you include the Remote Data Agent server Public IP and Private Ip addresses.
openssl x509 -req -extensions v3_ca \
-extfile <(echo "[v3_ca]"; echo "extendedKeyUsage=serverAuth"; echo "subjectAltName=DNS:${REMOTE_SERVER},DNS:*.${REMOTE_SERVER},IP:${REMOTE_SERV_PUBIP},IP:${REMOTE_SERV_PVTIP}") \
-days 820 -in ~/certs/rda.csr -CAkey ~/certs/ca_key.pem -CA ~/certs/ca_cert.pem -out ~/certs/rda.pem -passin pass:mypass
Create a certificate container file.
openssl pkcs12 -export -out ~/certs/certificate.pfx -inkey ~/certs/rda.key -in ~/certs/rda.pem \
-certfile ~/certs/ca_cert.pem -passout pass:mypass -passin pass:mypass
Extract the private key and Certificate from container file.
openssl pkcs12 -in ~/certs/certificate.pfx -nocerts -out ~/certs/mykey.pem -nodes -passin pass:mypass
openssl pkcs12 -in ~/certs/certificate.pfx -nokeys -out ~/certs/chain.pem -passin pass:mypass
At the end of this process, you obtain the mykey.pem and chain.pem file for Remote Data Agent Service configuration.
Create specific folder structure at Remote Data Agent Server
To run the Remote Data Agent services, it requires a specific folder structure owned by user:group 1001:1001. The required files like SSL Cert file, Viya Asset License file, Container Manager runtime, and Data Agent config filers needs to be placed under specific folder. In the following example the deploy folder name could be anything like “CDERemoteDA_Deploy” , but underneath subfolder name has to be same as listed here.
cd ~
mkdir CDERemoteDA_Deploy
cd CDERemoteDA_Deploy
sudo mkdir -m 700 sasdata
sudo mkdir -m 700 sasdata/data
sudo chown 1001:1001 -R sasdata/
Copy SSL Certificate to Remote Data Agent deploy folder
Copy/move the SSL certificate to Remote Data Agent deploy folder. You can have your own certificate or generate by using above steps described. Change the owner and group of files to 1001:1001.
cp ~/cert/mykey.pem ~/CDERemoteDA_Deploy/sasdata/mykey.pem
cp ~/cert/chain.pem ~/CDERemoteDA_Deploy/sasdata/chain.pem
sudo chown 1001:1001 -R ~/CDERemoteDA_Deploy/sasdata/Copy Viya Asset, License file, and Container Manager file to Remote Data Agent deploy folder
Copy/move the Viya asset, license, and container manager files to the Remote Data Agent deploy folder, files downloaded from my.sas.com site. I am assuming downloaded files are under ~/mydownload folder at Remote Data Agent Server. The following is the example asset file and license name. Notice the folder location is different from last step.
cp ~/mydownload/SASViyaV4_9CV11D_certs.zip ~/CDERemoteDA_Deploy/
cp ~/mydownload/SASViyaV4_9CV11D_license.jwt ~/CDERemoteDA_Deploy/
cp ~/mydownload/containermgr-linux.tgz ~/CDERemoteDA_Deploy/Deploy Remote Data Agent container image by using container manager
The container manager will deploy the Remote Data Agent container image from cr.sas.com to on-premises server. If you have a local container image repositor, you can use e.g. --image-repository repulpmaster.unx.sas.com ; to deploy the container images. The deployment process use the SAS Viya asset file to fetch the compatible software.
cd ~/CDERemoteDA_Deploy/
tar xvf containermgr-linux.tgz
./container-manager install --deployment-data SASViyaV4_9CV11D_certs.zip sas-data-agent-server-remote
Copy and update Remote Data Agent properties files to deploy folder
When you run container manager to deploy Remote Data Agent container image, it also provide a sample configuration file. You copy these files and update as per your environment. The sample config files are located under ~/.sas-container-manager/sas-data-agent-server-remote/stable/files/.
cd ~/CDERemoteDA_Deploy
cp ~/.sas-container-manager/sas-data-agent-server-remote/stable/files/* .
Update da-vars.env file
The da-vars.env file is meant for system level configuration of Remote Data Agent. The configuration file includes variable like corresponding Viya Service URL and OAuth secret used at co-lcoated data agent configuration. The Secret must match with the Co-located Data Agent configuration parameter. This config file also contains the location and file name of SSL certificate and key for Remote Data Agent server.
Following is the example from da-vars.env.
# update the SAS_DA_SERVICES_URL= from your Viya Platform URL. The listed value is just as an example.
SAS_DA_SERVICES_URL=https://utkuma-p03089-rg.gelenable.sas.com:443
# The SAS_DA_OAUTH_SECRET= hasa the same value what you have it for CDA configuration file at Viya Platform .
# The Cert file is the copy of same file used at Viya platform
SAS_DA_NAMESPACE=gelenv
SAS_DA_OAUTH_SECRET=*3434F0D512A9E05E4EF36450FC676EF151D94B51
SAS_DA_SSLCALISTLOC=/sasdata/chain.pem
SAS_DA_SSLPVTKEYLOC=/sasdata/mykey.pem
SAS_DA_SSLCERTLOC=/sasdata/chain.pem
# Update SAS_DA_RESTRICT_CONTENT_ROOT= with value as FALSE , which will allow data access from folder other than /data folder.
# By default RDA can read and write data files to /sasdata/data/ CDE deploy folder.By Putting "FALSE" you can use other folder as well.
SAS_DA_RESTRICT_CONTENT_ROOT=FALSE
Start Remote Data Agent Container Service
Copy/move the Viya asset, license, and container manager files to the Remote Data Agent deploy folder, files downloaded from my.sas.com site. I am assuming downloaded files are under ~/mydownload folder at Remote Data Agent Server. The following is the example asset file and license name. Notice the folder location is different from last step.
cd ~/CDERemoteDA_Deploy
./container-manager start \
--license-path SASViyaV4_9CV11D_license.jwt \
--sasdata sasdata \
--vars da-vars.env \
--data sasdata/data \
--access-vars sas-access.properties \
sas-data-agent-server-remote
Verify Remote Data Agent service log
You can view and verify the log of Remote Data Agent service by using docker statement. The log includes all the steps performed to start the Remote Data agent service.
jumpuser@p03089-jump-vm:~/CDERemoteDA_Deploy$ docker logs sas-data-agent-server-remote
{"level":"info","message":"Starting Vault....","properties":{"caller":"sas-data-agent-server-remote-entrypoint.sh:104","logger":"Script","pod":"sas-data-agent-server-remote","sessionUser":"Script:sas","thread":"00000000"},"source":"dagentsrv","timeStamp":"2023-05-10T16:22:39.421542+00:00","version":1}
.....
...............
...............
{"level":"info","message":"SAH061999I Server SAS Data Agent, State, running","properties":{"caller":"tkecrapp.c:270","logger":"Admin.Operations","pod":"sas-data-agent-server-remote","sessionUser":"Server:sas","thread":"00000005"},"source":"dagentsrv","timeStamp":"2023-05-10T16:22:59.704000+00:00","version":1}
jumpuser@p03089-jump-vm:~/CDERemoteDA_Deploy$
Verify the network access between Remote Data Agent and Viya Platform
Once Remote Data Agent container service is up and running at on-premises server. You should verify the access path between Remote Data Agent and Viya platform. The verification process will prove whether SSL certificate is working or not with port open at on-premises firewall. The verification steps include the execution of curl statement at SAS Viya pod container to access Remote data agent container and vice -versa.
At Viya Platform hop onto data agent container and execute Curl Statement to Remote Data Agent container.
[cloud-user@pdcesx03089 ~]$ kubectl get pods -n gelenv | grep data-agent
sas-data-agent-server-colocated-0 1/1 Running 0 99m
sas-data-agent-services-d77bcf87c-jxr5v 1/1 Running 0 100m
[cloud-user@pdcesx03089 ~]$
cloud-user@pdcesx03089 ~]$
[cloud-user@pdcesx03089 ~]$ kubectl exec -i -t -n gelenv sas-data-agent-services-d77bcf87c-jxr5v -c sas-data-agent-services -- sh -c "(bash || ash || sh)"
bash-4.4$
## Once you are in the Pod’s shell, run the curl statement with Remote Data Agent public/pvtIp and port.
bash-4.4$ curl -kv https://192.168.2.5:25141
* Rebuilt URL to: https://192.168.2.5:25141/
* Trying 192.168.2.5...
* TCP_NODELAY set
* Connected to 192.168.2.5 (192.168.2.5) port 25141 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
....
..........
At Remote Data Agent server, hop onto Remote Data Agent container and execute Curl Statement to access SAS Viya platform (SAS Viya Service URL). You need to update the AKS cluster Security Network Group (NSG) to include the incoming traffic from Remote Data agent Ips.
jumpuser@p03173-jump-vm:~/CDERemoteDA_Deploy$ docker exec -it sas-data-agent-server-remote bash
bash-4.4$
bash-4.4$ curl -kv https://utkuma-p03173-rg.gelenable.sas.com:443
* Rebuilt URL to: https://utkuma-p03173-rg.gelenable.sas.com:443/
* Trying 20.241.132.184...
* TCP_NODELAY set
* Connected to utkuma-p03173-rg.gelenable.sas.com (20.241.132.184) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
.....
.............
........................
Register the Remote Data Agent server to Viya Platform
With successful access path verification between Remote Data Agent Server and Viya platform your Viya environment is ready to register a Remote Data Agent server. The Data Agent registration enables SAS compute server and CAS to access the data located at on-premises server using Remote Data Agent Data Sources.
The sas-viya CLI is the interface to administer CDE Data Agent Service. Before you can use the CLI you need to authenticated against the SAS Service URL .
[cloud-user@pdcesx03089 ~]$ export SSL_CERT_FILE=~/.certs/my_trustedcerts.pem
[cloud-user@pdcesx03089 ~]$ sas-viya auth login --user geldmui@gelenable.sas.com --password 'XXXXXXXXXXX'
Enter credentials for https://utkuma-p03089-rg.gelenable.sas.com:
Login succeeded. Token saved.
[cloud-user@pdcesx03089 ~]$
While registering, use the Public Ip (or Pvt Ip) address of Remote Data Agent server along with the Port # which is open at on-premises Firewall.
sas-viya dagentsrv servers register --name sas-data-agent-server-remote --address https://192.168.2.5:25141 --context-root rda
Execute few CLI statement to access the Remote Data Agent components. The successful execution of these statement verifies that the environment configuration is good and it’s ready for various data source ( SAS datasets, Oracle etc… ).
[cloud-user@pdcesx03089 ~]$ sas-viya profile set-output text
[cloud-user@pdcesx03089 ~]$ sas-viya dagentsrv servers set-default --data-agent sas-data-agent-server-remote
The default data agent server was successfully set.
[cloud-user@pdcesx03089 ~]$ sas-viya dagentsrv server list
Name Host Port Attributes
sas-data-agent-server-colocated utkuma-p03089-rg.gelenable.sas.com 443 contextroot=dataAgentServerColocated;ssl=Y
sas-data-agent-server-remote 192.168.2.5 25141 contextroot=rda;ssl=Y
[cloud-user@pdcesx03089 ~]$ sas-viya -k dagentsrv data-sources list
Name ID Type
ADMIN ADMIN DSN
BASE BASE DSN
BASE BASE__DATA_SERVICE__ Service
[cloud-user@pdcesx03089 ~]$ sas-viya -k dagentsrv services list
Name Type Domain Version Options
BASE base --- 2.5 ---
__SERVER__ server --- 2.5 PURGE_CACHE=30;CASE_SENSITIVITY=(OBJECT=F;COLUMN=F);CACHE=(NAME=AS;TIMEOUT=300)
Next, stay tuned-in for post on various data source configuration at Remote Data Agent server .
Many thanks to Brian Hess for sharing the expert knowledge and help on this post.
Important Links:
Cloud Data Exchange for the SAS Viya Platform
SAS Viya Cloud Data Exchange Deployment and Configuration (Part -1)
