BookmarkSubscribeRSS Feed

SAS Viya Cloud Data Exchange Deployment and Configuration (Part -1)

Started ‎06-22-2023 by
Modified ‎08-16-2023 by
Views 1,408

In my last post I talked about SAS Viya Cloud Data Exchange, it’s components and how it can help SAS users to access on-remises data from cloud hosted SAS Viya platform. As mentioned in last post, the configuration between Remote Data Agent and SAS Viya platform is a challenging task. So, let’s look at the details steps to integrate Remote Data Agent service with SAS Viya platform.

 

This post is in two parts, the first part describing the deployment and configuration of Co-located Data Agent service. The second part is describing the deployment and configuration of Remote Data Agent.

 

The SAS Viya Cloud Data Exchange software is available with most of the SAS Viya bundle license. There is no separate license for Cloud Data Exchange components like Co-located Data Agent service and Remote Data Agent services.

 

SAS Viya Cloud Data exchange have two types of Data Agents.

  • Co-located Data Agent - Reside at SAS Viya platform environment.
  • Remote Data Agent - Hosted at on-premises data center.

 

uk_1_SASViya4_CDE_1.png

 

 

Co-located Data Agent Deployment and configuration

 

The Co-located Data Agent software and components are delivered with the standard SAS Viya deployment. The Co-located Data Agent (CDA) pods and services starts along with the rest of SAS Viya services. Configuration of Co-located Data Agents are described in the following steps.

 

Use example configuration file for Co-located Data Agent (CDA)

 

The CDA configuration example is provided under ../sas-bases/examples/sas-data-agent-server-colocated folder to configure the Co-located Data Agent and integrate with Remote Data Agent services. You can copy the example folder to site-config folder and update the configuration files.

 

cp -r …./sas-bases/examples/sas-data-agent-server-colocated …./site-config/
chmod 755 …./site-config/sas-data-agent-server-colocated/*.*

 

There are two configuration files sas-data-agent-server-colocated-secret.properties and sas-data-agent-server-colocated-config.properties.

 

The sas-data-agent-server-colocated-secret.properties file contains the SAS_DA_OAUTH_SECRET= variable. The value of SAS_DA_OAUTH_SECRET= is used by Remote Data Agent service to connect to SAS Viya platform for system level information like SAS Viya user identity. You must provide the same value while configuring Remote Data Agent service at on-premise server. The password could be in plain text or encrypted or hashed.

 

# hashed password in sas-data-agent-server-colocated-secret.properties
SAS_DA_OAUTH_SECRET=*3434F0D512A9E05E4EF36450FC676EF151D94B51

 

The sas-data-agent-server-colocated-config.properties file is meant for Data Agent level configuration. It contain variable like debugging trace flag, user data folder, SAS Access clients variables and so on.

 

# Variable examples in  sas-data-agent-server-colocated-config.properties 
SAS_DA_DEBUG_LOGTYPE=TRACEALL
SAS_DA_RESTRICT_CONTENT_ROOT=TRUE
SAS_DA_CONTENT_ROOT=/gelcontent/data/

Update kustomization.yaml file

 

Include the sas-data-agent-server-colocated-secret.properties and sas-data-agent-server-colocated-config.properties file in kustomization.yaml to build and start the Co-located Data agent with updated parameters.

 

printf "
- command: update
  path: secretGenerator[+]
  value:
    name: sas-data-agent-server-colocated-secrets
    behavior: merge
    envs:
      - site-config/sas-data-agent-server-colocated/sas-data-agent-server-colocated-secret.properties" | yq -I 4 w -i -s - …./kustomization.yaml


printf "
- command: update
  path: configMapGenerator[+]
  value:
    name: sas-data-agent-server-colocated-config
    behavior: merge
    envs:
      - site-config/sas-data-agent-server-colocated/sas-data-agent-server-colocated-config.properties" | yq -I 4 w -i -s - ….kustomization.yaml

 

Build and deploy the updated manifestation

 

cd ~/project/deploy/
kustomize build -o site.yaml

NS=myenv;
kubectl apply -n ${NS} -f ~/project/deploy/gelenv/site.yaml ;

## Recycle Data Agent Pods.
kubectl -n ${NS} delete pod --selector='app.kubernetes.io/name=sas-data-agent-server-colocated'
kubectl -n ${NS} delete pod --selector='app.kubernetes.io/name=sas-data-agent-services'

 

Add SAS Viya users to DataAgentAdmin and DataAgentPower user group

 

The administrative tasks of Data Agents are executed using sas-viya CLI. Make sure you have sas-viya CLI installed at your local client machine or at jump server. To execute Data Agent administrative task, user must be in a specific SAS Viya user group. The users from DataAgentAdminstrators group can configure and administer Data Agents, Data Source , Security Domain and Data Services. The users from DataAgentPowerUsers group can access the data sources by executing DQL, DML and DDL statement. As per user place them in the respective SAS Viya user’s group.

 

export SSL_CERT_FILE=~/.cerrts/my_trustedcerts.pem
sas-viya auth login --user sasboot –password

sas-viya identities add-member --group-id DataAgentAdministrators --user-member-id geldmui@gelenable.sas.com
sas-viya identities add-member --group-id DataAgentPowerUsers --user-member-id geldmui@gelenable.sas.com

sas-viya identities add-member --group-id DataAgentPowerUsers --user-member-id gatedemo001@gelenable.sas.com

sas-viya identities add-member --group-id DataAgentPowerUsers --user-member-id gatedemo002@gelenable.sas.com

 

With the above steps, SAS Viya environment is configured to integrate with Remote Data Agent services. You can also create data sources at Co-located Data Agent server which are directly accessible from Viya platform.

 

Stay tuned-in for second part of this series on Remote Data Agent Deployment and Configuration.

 

Many thanks to my colleague Brian Hess for sharing the expert knowledge and help on this post.

 

Important Links:

 

Document: Cloud Data Exchange for the SAS Viya Platform

Blog: Cloud Data Exchange for the SAS Viya Platform    

Comments

I have a question on updating CDE as part of the stable/LTS process. I could not find specific references to updating CDE in the documentation. How often is CDE updated (I'm assuming it could follow the stable/LTS cadence)? And what is the process to update it (of course without having to recreate data agents)?

 

Hi, 

 

Let’s break CDE into the CDA (Co-located Data Agent ) and the RDA (Remote Data Agent).

 

For the CDA, nothing to do!  The CDA will detect software updates and handle it.  The CDA k8s data will get updated, and eventually (according to Kubernetes behavior) the CDA pods will get restarted.

 

For the RDA, we have documentation on software updates here:
https://go.documentation.sas.com/doc/en/pgmsascdc/v_041/dplydagent0ctr/p09x6g5gu8jcv3n0zg7ok6n9gx0d....

 

The customer is in control for when they accept a new RDA image.

 

-Uttam 

Thank you, Uttam - what about the configured data agents? Is Postgres part of the container image or would this be outside the container?

Hi, 

 

The data agent configuration is stored in two places SYSCAT ( PGDB)  and Vault . As long as you have backup of the SYSCAT and Vault , you should be able to get the configuration back by restoring from backup. 

 

The Postgres server is part of Remote Data Agent (RDA) container, but data files are outside of the container. The Data files reside on filesystem (data ) mounted to RDA. 

 

In general when you upgrade to new RDA container, The RDA will use the existing PGDB foot print and  Volt  to start the RDA with configurations. 

 

The Postgres server upgrade inside the container should be transparent to the user, i.e. PG shouldn’t break its customers!  In case there is major upgrade on PGDB side (like PG15 or so ) we will have documentation to the RDA guide to indicate whatever the customer will need to do as part of the upgrade.  The same holds true for Vault. 

Version history
Last update:
‎08-16-2023 10:38 AM
Updated by:

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags