With the July version of SAS Viya (stable 2024.07), a nice change has been made in the section of the SAS Viya Operations guide that covers the customizations of a deployment.   Before this version, the "Common Customizations" page just provided a long list of all the customizations (outside of product-specific customizations) that should be considered for a deployment.   But now this "Common Customizations" content has been divided in two parts :   Required customizations Optional customizations   While many customizations are optional, a few of them should be decided in advance (and they sometimes can’t be made without specific pre-requisites in place).   This documentation change helps to identify some of the key decisions that MUST be made (in a way that corresponds to the customer needs or requirements) before starting any new Viya deployment.   So, I thought it would be great opportunity to revisit and discuss these architecture decisions and deployment pre-requisites through the review of these six "Required customizations" and explain why they would form the basis of a Viya environment Architecture design document.   As we often say during our GEL workshops:   "The deployment of SAS Viya by itself is usually a matter of hours (or even minutes). However, the discussions, decisions and preparation leading to that moment can take days or even weeks..."    TLS Mode Host and port for Cluster Ingress CAS topology and workload strategy Internal or external PostgreSQL?  Internal or external OpenSearch? StorageClass to use for RWX PVC ? "Recap" table   Note: In this post we don't explain HOW these customizations are implemented as it is already nicely explained in our documentation or in the README files.     TLS mode ?   TLS (Transport Layer Security) allows to encrypt data in motion and is used in the SAS Viya platform to secure communications with internal and external components.   3 options are possible : "Full stack TLS", "Front door TLS", or "no TLS".   By default (using the kustomization.yaml example file provided in the documentation), your Viya environment will be automatically configured with "Full stack TLS". It means that all the microservices and web applications implement HTTPS in their communications. In this mode, all network connections from outside and even inside the Viya environment are secured.   With "Front door TLS", only the components that are intended to accept connections from outside the Kubernetes cluster (such as the Ingress controller and CAS server) are configured to only accept encrypted network connections.   In both cases, the customer can use the default certificate issuer (openSSL) or use another certificate issuer (cert-manager). The customer can also provide their own certificates if they prefer to secure the ingress.   The customer should always be advised to use TLS to secure the network traffic ("No TLS" should never be used as the traffic would be in clear text and easily intercepted by anyone). However, most of the time, customers are very sensitive about  security and opt for the "Full stack TLS option".   Reference :   SAS Doc : https://go.documentation.sas.com/doc/en/itopscdc/default/itopssr/n18dxcft030ccfn1ws2mujww1fav.htm#n0w1os9ufnn7xwn1an3qbhxjxze9     Host and port for Cluster Ingress As part of the deployment, you need to provide a small set of parameter’s values in your initial kustomization.yaml file.   In the officially documented example, they are represented as place holders under this form:  {{ PLACE_HOLDER }}. One of them is the {{ NAME-OF-INGRESS-HOST }}   How can you find this value ?   I wrote a couple of posts (part 1 and 2) to answer this question, here is an extract :   {{ NAME_OF_INGRESS_HOST }} corresponds to the URL base that the customer wants to expose to the SAS applications users. The {{ PORT }} parameter is optional, as the HTTP schema is usually determined by the start of the url (“http” or “https”). However, the important thing to understand here is that {{ NAME_OF_INGRESS_HOST }} also correspond to a DNS name that must be associated to the external IP address exposed by the Ingress controller.   When you install the Ingress controller in a Cloud environment, an external IP address is automatically provisioned by the Cloud provider (You can see this address in the EXTERNAL-IP column when you describe the ingress controller service).   A DNS name then needs to be mapped to the ingress public IP by the site’s IT team configuring their DNS services. The exact components of the FQDN will likely be shaped by the site’s DNS standards, for example: https://viya.your_dept.your_site.com/ and Viya accommodates that.   But there must be a DNS entry associated with this IP address. It is a key pre-requisites that should not be overlooked during the deployment planning process.     CAS topology and workload strategy   As the main analytics and data management engine of the SAS Viya platform, CAS is a cornerstone of the environment. A CAS Server is required at the start-up time of Viya : it is a pre-requisite to have at least one CAS Server defined otherwise the Viya platform will not work for any user (whether the users plan on using CAS or not).   Some choices must be made at a minimum for the CAS topology and the CAS workload strategy.   While, by default (using the kustomization.yaml example file provided in the documentation) the CAS server consists of a Single-node (SMP : for Symmetric Multi-Processing), it is possible to opt for a Multi-node CAS server (MPP: for Massively Parallel Processing) to spread the Analytics processing and data across multiple CAS nodes.   This choice is mostly driven by the amount of data to process (remember that with CAS, the data are loaded into memory to be processed by CAS, and a single machine may not have enough RAM to hold them) and an official sizing is a key input in this decision.   Note that more sophisticated CAS topologies are also possible such as:   Adding a secondary CAS controller into a CAS MPP deployment (to improve the CAS Server availability) Adding new CAS server(s), for example to isolate and specialize the different kind of workloads (ex: 1 CAS server for the reports, 1 other for the Visual forecasting, etc…)   Now, regarding the CAS workload strategies, several options are also possible. The SAS general recommendation is to choose the default "auto-resourcing" configuration. This approach ensures that each CAS instance (Controllers and workers) is running on a dedicated host and can leverage all its capacity (in a similar way as with a Viya 3.x deployment). While this is ideal for maximum performance, it might be overkill for dev/test environments.     Note that options are also available to have a differentiated workload classes for the CAS controller(s) and the workers. The goal is to optimize the costs by using distinct node pools and smaller machines for the CAS controller(s) that usually don’t require as much CPU and memory as the workers.   The CAS topology and workload management topics are a key part of the Architecture design and should be discussed with your site’s IT team to make sure the associated decisions meets your site’s needs and constraints.   Another exciting benefit of running Viya 4 as containerized workloads in a Kubernetes cluster on virtualized hardware is that changing your deployment topology can be effected simply by submitting configuration changes programmatically. This should give your site’s administrators confidence to try new workload configurations in pursuit of best performance, efficiency, and cost.   If these concepts are not clear to you, the  SAS Viya Architecture course has dedicated modules to cover them.   References :   CAS Server Topology Changes And CAS Table Balancing:  https://communities.sas.com/t5/SAS-Communities-Library/CAS-Server-Topology-Changes-And-CAS-Table-Balancing/ta-p/903549       Internal or external PostgreSQL?   The SAS Viya platform relies on an instance of PostgreSQL for its SAS Infrastructure Data Server. This instance can be either internal or external.   By default, SAS Viya is installed with the internal PostgreSQL instance that relies on Crunchy data server which is shipped with the SAS order. But the SAS Viya platform can also be configured to use an external instance of PostgreSQL such as the Azure PostgreSQL Flexible Server.   This post, from my colleague Edoardo Riva, lists some of the considerations associated to the choice of an internal versus external instance.   It is worth noting that many SAS Solutions also require an additional PostgreSQL server instance for the Common Data Store (aka CDS), which requires its own PostgreSQL instance. When deploying a SAS Solution, the location of the Common Data Store instance (internal or external) - as well as any additional potential instances, for Airflow for example - must also be discussed and addressed.     Internal or external OpenSearch?   The SAS Viya platform also relies on an instance of OpenSearch to create and store search indices (for example for the SAS Information Catalog and Visual Investigator).   Starting with 2023.03 it is possible to use an external instance of OpenSearch.   Once again, you can find a nice post from Edoardo, comparing the use of an internal versus an external instance of OpenSearch.   The decision between : letting the SAS Viya deployment install and manage OpenSearch (default) versus using an existing external instance, must be made before the deployment.     StorageClass to use for RWX PVC ?   The last item of the "Required customizations" covers another key decision : which kind of storage to use for the Viya component that requires a ReadWriteMany (aka RWX) storage class.   As a reminder, the RWX storage type can be seen as the equivalent of a shared file system in the bare OS world. It corresponds to a kind of storage that supports concurrent access by multiple pods running on different nodes.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Since the SAS Viya PersistentVolumeClaims don’t specify a StorageClassName, the default storage class specified by your Kubernetes configuration is used.   This  is convenient because most of the Managed Kubernetes platforms that we support in the Cloud come with a default storage class.  On the other hand, it’s not often the ideal choice because the usual default storage class is backed up by managed disks, which has only RWO access(standard-rwo in GCP, managed-csi in Azure, etc..).   While many of the SAS Viya platform volumes only need storage with RWO access, some of them require RWX storage…As a consequence, if you don’t do anything regarding the storage class when deploying for example in AKS, GKE or EKS, your deployment will fail.   So, for them we need to provide an alternative RWX storage class and make the associated configuration changes, as in the example below :     See how the annotationSelector is used to target the PVC that requires this RWX access.   So, the associated architecture decision is: "What storage class should I use for the Viya components that require RWX access ?"   Several options are possible, they depends on the Kubernetes platform. You can have a look at this table from the Kubernetes documentation to get a first list of RWX compatible volumes plug-ins, but options mostly depends on the Cloud provider.   Note that if you are using the viya4-iac-x (IaC) or  viya4-deployment (aka DaC) tools, you don’t have to tweak the Kustomize Transformer files : the DaC automatically configures a new RWX storage class and make it the default. In this case, the new storage class is backed up by an NFS Client provisioner pointing to either a Cloud Managed File services (such as Azure NetApp, GCP File store or EFS in AWS) or to an NFS server running on a Cloud VM (depending on your IaC storage choice).   The available DaC options may correspond to the customer’s needs and requirements. Otherwise, it is required to identify a RWX storage class and make the configuration changes so it is used for the Viya volumes that requires it. Note that you can also use distinct RWO and RWX storage classes for the different volumes required by the Viya platform.     Other key decisions   Finally, it is important to remember that there several  key decisions that are NOT reflected by a specific requirement listed in the new "Required Customization" section.   Some of them (Kubernetes platform, dedicated or shared cluster, etc…) are critical and early to decide, with downstream implications on the other aspects of the architecture and deployment (networking, storage, etc…).   While it is probably not an exhaustive list, the decisions listed below correspond to the customer’s specific requirements or constraints and must also be made during the design phase (before the deployment) :   Kubernetes Platform ? AKS? GKE? EKS ? OpenShift ? other ? Dedicated or shared Kubernetes cluster ? Cadence (LTS or stable) ? Storage : used for the Viya platform components and for things like input/output data, user’s home directory, etc…as an example, for production environments, block storage (instead of NFS based storage) is strongly recommended for specific SAS Viya components (OpenSearch, Redis and RabbitMQ) volumes ! Network : kubelet or CNI ? (Note that the question is not relevant for deployment of SAS Viya on Upstream Opensource Kubernetes cluster, where only the Calico CNI is supported.) Identity provider and authentication mechanism : various options are available for the integration with the Identity provider (LDAP or SCIM) and for the authentication.   We have added this decisions into our recap table below.     Recap and advice   The table below lists the key decisions that should be made before performing the initial deployment with their default value, pre-requisites and discuss whether they could be changed after the initial deployment.     Decision Default value Pre-requisites Can be changed  after the initial deployment? TLS mode “Full-stack” TLS Custom certificates (in .pem format) and the associated private key must be provided if the customer wants to use their own certificates for the ingress (instead of the default ones). Yes. The TLS mode can be changed afterwards, however it is recommended to agree on a TLS mode before the deployment to avoid any outage. Ingress host name and DNS None The customer needs to associate a DNS name to the Ingress controller service external IP , available Yes. However if the  INGRESS_HOST is wrong, nobody will be able to login to the deployed environment from the outside. CAS Topology and Resource Management strategy SMP, CAS-autoresourcing None Yes. The topology and workload placement strategy can be changed but it will impact the deployment and users (outage and need to reload the data). Internal or External PostgreSQL instance(s)? Internal None with the internal instance. For external: extensions, number of connection, pre-created SharedServices database, etc…see this page Yes but only in one way. SAS has created the PostgreSQL Data Transfer Tool for the SAS Viya Platform to move content from the internal instance of PostgreSQL to an external instance. But it is not possible to change from external to internal. Internal or External OpenSearch instance? Internal   No. Be careful, after the deployment has completed, you cannot modify your deployment to use a different OpenSearch instance. RWX Storage class to use for the SAS Viya Persistent Volumes ? There is usually no default RWX storage class available in the cluster. But the IaC tool for opensource Kubernetes and the DaC tool creates one for the SAS Viya volumes. A RWX storage class must be made available otherwise the deployment will fail. Maybe but not easily…while some volumes can be rebuilt from scratch, other may hold user’s data that needs to be migrated to new storage devices. Kubernetes platform ? None A supported Kubernetes platform should me made available with cluster-wide privileges No. If the Kubernetes platform is changed, SAS Viya must be redeployed (however a full-system migration could be done from one Cloud provider to another). Dedicated or shared Kubernetes cluster ? NA When multiple Viya versions are sharing the same cluster, there are some caveats to be aware of. Keeping all the Viya environments at the same version is strongly recommended. See above. LTS or stable cadence ? None To remain fully supported an update of the version N of SAS Viya must be made before the version N+4 is released. It means that customers on the stable cadence must update their Viya at least version every 3 to 4 months to avoid to be in limited support. Yes it is possible to change the cadence (under certain conditions). Storage strategy for the platform data and the user’s data NA Storage classes or PV types to use for the Viya platform volumes as well as input/output data must be made available Possible but requires tests, backup, outage and migration of the content. Network type ? kubelet ? CNI As of today, kubenet is usually the default for Managed Kubernetes Services in the cloud. In some cases (SAS with SingleStore, deployment with open-source Kubernetes) a CNI plugin is required. So always check the latest documentation version for requirements. Changing from one CNI implementation to another may be possible depending on the Cloud provider but requires tests, backup, outage and migration of the content. Refer to the specific cloud provider’s documentation. Identity provider and authentication mechanism NA The requirements depends on the type of identity provider and authentication, see the official documentation The identity provider cannot be changed for an “in-use” system. authentication mechanism can be changed in some cases but could require an outage and rebuild and re-apply the deployment manifests   My final advice would be to make sure that you have captured at minima the architecture decisions listed in this table (and prepared the associated pre-requisites) before starting your SAS Viya deployment process.   Also keep in mind that some post-deployment tasks are usually required as well before users can start to access the SAS Viya applications. They also depend on the offering and products that are deployed. See this page in the official documentation for a list of post-deployment steps.   Hope that helped ! Thanks for reading 😊     Find more articles from SAS Global Enablement and Learning here. ... View more

As a follow-up of my previous post "Manage your SAS Viya platform configuration in a Git Repository", I wondered if it would be possible to store and manage the SAS Viya configuration in a Cloud Provider’s Git repository service (instead of using something like GitLab or GitHub).   Git is the most widely used source-code management tool and Git repositories have become a key component of modern DevOps platforms. Azure, AWS and Google Cloud Platform offer "git compatible" Code Source repository services, such as Azure Repos, AWS CodeCommit or Google Cloud Source Repositories.   But can the SAS Viya platform play nicely with them ?   Here is a diagram of the generic deployment system that we’ll aim to put in place.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Assuming we use the "sas-orchestration command" deployment method and already have a Cloud Source Repository in place (created from the Cloud portal or with the CLI), our goal is :   To push an initial SAS Viya configuration (a folder containing the kustomization.yaml file and the site-config folder) to the Cloud Git Repository. Then to run the sas-orchestration command to deploy SAS Viya using the configuration files stored in the Cloud Git Repository.   The purpose of this post is to demonstrate, in a few steps, how to do that with the Google Cloud Source Repositories service.   Introducing the Google Cloud Source Repositories Git local authentication setup  Cloud Source Repository setup Prepare the initial SAS Viya configuration in the local Git repository Push the initial SAS Viya configuration to the Cloud Git Repository Edit your code in the Google CSR Deploying SAS Viya Conclusion   Introducing the Google Cloud Source Repositories   Google Cloud Source Repositories (or "Google CSR" to keep it short) is the Git Repository service available in the Google Cloud Platform market place*.   It has several interesting features (as you can see on the screenshot) and is a "fully-managed" Git repository.                     Google CSR currently supports 3 types of local authentication  :   SSH : Consists of a private key that resides on your local system and a public key that you register with Google Cloud. When using this mode, you’ll have to use your private SSH key to authenticate yourself against the Git repository. Google Cloud CLI : Cloud Source Repositories lets you authenticate by using the google cloud CLI (gcloud). You need to run the gcloud init command to authenticate in your GCP project. Manually generated credentials, a script is generated in the Google portal to create the .gitconfig and .gitcookies files (using the Netscape/Mozilla cookie file format), which is passed in the in the Git HTTP session, when the git command is used.   Google also documents 3 distinct kind of URLs to use in order to "git clone" the repository, corresponding to the authentication systems :   Cloning using SSH:   git clone ssh://[EMAIL]@source.developers.google.com:2022/p/[PROJECT_ID]/r/[REPO_NAME]   Cloning using the gcloud command:   gcloud source repos clone [REPO_NAME] --project=[PROJECT_NAME]   Cloning using the manually generated credentials:         When I started my research, I was not too sure which kind of local authentication to choose for my need to make the "sas-orchestration" tool read the SAS Viya configuration from the Google CSR.   However, with some hindsight, knowing that the sas-orchestration container relies on the "go-getter" library and requires that the "go-getter" protocol must be “Git over http(s)”, I could have been able to determine which authentication method to use…   (*) Note : Important: Effective June 17, 2024, Cloud Source Repositories isn't available to new customers. Organizations that have used Cloud Source Repositories prior to June 17, 2024 are not affected by this change. Git local authentication setup ....   End of the suspense 😊   The authentication method to choose here is the "Manually generated credentials".   It is a choice by elimination: the “Git over SSH” protocol is not supported for the DaaS tools go-getter URLs and the gcloud CLI is not included in the sas-orchestration image. When you opt for this method, you need to click on a link, which (after authentication with your Google account), generates a page as below: This page gives you the snippets of code to run to configure your local git client (for Windows and Linux). When you run this code locally, it creates an http.cookiefile and references it in the Git configuration (~/.gitconfig file). In a Linux operating system, it looks like this: This is how Git can authenticate you over http(s) to the source.developers.google.com domain. Once it is in place, you can run your git clone command to pull the content from your Google CSR.   Cloud Source Repository setup It is very easy to create your first CSR repository in Google, you can use the web interface (go to https://source.cloud.google.com/repos and click on "Add repository") or the gcloud CLI with a command like:   Prepare the initial SAS Viya configuration in the local Git repository.   gcloud source repos create raphpoumar-source-repo   Now, imagine that you have your initial set of SAS Viya configuration files, locally available on your jumphost/bastion linux machine, and you want to move them to your brand-new Google Cloud Source Repo. The following actions need to happen:   Clone the empty CSR repository with git Example:     It creates a local folder on your machine with the same name as your CSR repository. This new folder contains the git database (under the .git folder) is your local Git Repository.   Setup the local Git configuration (~/.gitconfig). Example:   #Setup git default identity git config --global user.email $EMAIL git config --global user.name $STUDENT git config --global credential.helper store git config --global http.sslVerify false   Create a local sub directory inside the local git directory and place your SAS Viya configuration files there. It would look like this:   Now add and commit the SAS Viya configuration files in the local git repository. Note that the commit message appears in the Google CSR version history after the push. At this point, the Git repository has been initiated but it remains local.   Push the initial SAS Viya configuration to the Cloud Git Repository.   It is time to register the remote Git repository (which is in the Google CSR) and to push our local changes there.   First, we add our local repository as a remote:   STUDENT=$(cat ~/student.txt) PROJECT_NAME=sas-gelsandbox git remote add googlehttps://source.developers.google.com/p/$PROJECT_NAME/r/$REPO_NAME   And then we can push from your local Git repository to the Google Cloud repository But then…you might see this kind of message: With a worrying line at the end saying that there was an error to push our files 🙁   Now, if we run the "git describe" command provided in this message, we see that the culprit is the file site-config/postgres/ServiceAccountKey.json. What the message actually means, is that our push has been rejected because the service has detected that we were trying to upload what looks like a private key (the "ServiceAccountKey.json" file) ! In a standard usage of the Git repository, it is indeed a bad idea. We usually don't want to place any credential information in a code repository. However, our case is a bit specific as we are not keeping source code but configuration files. In addition, we need this key to configure PostgreSQL and our Code Source repository is secured and private, so we can override this default security setting with the nokeycheck option. So, let’s try again   git push -o nokeycheck --all google Now we should see something like that: The problem is fixed 😊 If we open the Cloud Source Repositories web page, we can confirm that our configuration files have been transferred !   Edit your configuration files in the Google CSR   OK so now my kustomization.yaml and site-config folder files are in the Google source repository, but wait…how do I update them if I want to change my SAS Viya configuration? There are several ways. You can make the changes in your local git repository (from the local Linux machine used to do the first push) and add/commit/push them into the Google CSR remote repository. But it is also possible to edit directly your configuration files from the Cloud Source Repositories web UI. You’ll first need to give additional IAM permissions in GCP (like "Source Repository Writer"), but then you can click on the "Edit code" button and  a new Cloud shell instance will be started and the editor directly opens your configuration files (as shown below). Finally, even easier, if you have Visual Studio Code (with the Git plugin) installed on your local Windows workstation, you can also use it to modify your SAS Viya configuration files, commit and push the changes (as long as you have updated your Windows Git configuration to use the manually created credentials). So, now that our configuration files are in place and can be edited in the Google CSR, let’s see how to pull them when we deploy the software 😊   Deploying SAS Viya The final step is to set up the go-getter URL in the sas-orchestration command and run it to deploy SAS Viya! The parameter that tells the sas-orchestration container where to find the SAS Viya Configuration file is "user-content". Unlike with the GitLab example shown in my previous post, we cannot use a Git private token for the authentication (this authentication method was not available with the Google CSR). Instead, we need a way to make our credentials available for the sas-orchestration container. Here is how the command looks like:     When we activate the verbose mode of the sas-orchestration tool we can see that the git clone command is executed inside the container to clone the remote repository. So, as shown in the command example, the trick is to mount our .gitcookies and .gitconfig files inside the sas-orchestration container and to help Git to pick up the .gitconfig file by setting the $HOME environment variable. In this way the sas-orchestration tool has all the required credentials to perform the “git over http(s)” request to pull the viyaorch content from the Google repository.   Conclusion   Finally, we have seen that it is possible to run a Viya deployment from a configuration stored in the Google Cloud Source Repository. If you want to test it by yourself, a hands-on called "Deploy Viya 4 from Google Cloud Source Repositories" is available in the "course exercises" notes of the “SAS Viya 4: Deployment on Google Kubernetes Engine” GEL Lab: we’ll drive you through all the configuration steps described in this post. We could then imagine to use the Google CSR as the central location for all our SAS Viya environments configurations. As we've seen, the configuration files could be updated there in various ways and scenarios:   Directly edited from the Google CSR UI Spin up an ad hoc VM that "git clone" the repository, update a given configuration, then push the changes. Then another instance or process can run the deployments. Synchronized from GitHub or Bitbucket repositories The initial diagram was updated to reflect all these new opportunities 😊 Thanks for reading !   Find more articles from SAS Global Enablement and Learning here. ... View more

If you are about to deploy the latest SAS Viya version (2024.06 or later) and are using an external PostgreSQL server, there is a new system requirement.   If you miss it, your deployment may fail….This post explains the rationale and provides some guidance on how to meet this new requirement. We'll look at the "What?", "Why?" and "How?".   What?   From the "What’s new page" for SAS Viya stable 2024.06:   “Starting with 2024.06, the SharedServices database for an external PostgreSQL server is no longer created automatically during the initial deployment of the SAS Viya platform. Instead, you must manually create it before you start the SAS Viya platform deployment. This database supports the required SAS Infrastructure Data Server component."   In a SAS Viya deployment, the SAS Data Server Operator pod (aka "DSO") is the component responsible for creating the SharedServices database in the SAS Infrastructure Data Server (aka "IDS") instance.   From the stable 2024.06 version, the DSO does not create the SharedServices database if the PostgreSQL implementation is external.  The DSO still creates the SharedServices database if the Internal Crunchy Data Server is used.   Symptoms if you miss it   It is an important change.   If we forget about this requirement and deploy as we did prior to Stable 2024.06, we would probably see that our deployment fails with most pods either in "error" or stuck in the "Init: 0/x " status with a message like the one below from the sas-start-sequencer container’s log:   {"level":"info","version":1,"source":"start-sequencer","messageKey":"DB Check sleeping","messageParameters":{"elapsed time":"21m:29s:301ms","name":"main"},"properties": {"caller":"sdsci/main.go:405   In k9s, the pod’s view would look like that:     And after some troubleshooting, we would ultimately identify the root cause by looking in the SAS Data Server Operator pod’s log (the pod’s name is something like "sas-data-server-operator-c97c5c796-nxmnc") :   {"level":"error","version":1,"source":"sas-data-server-operator-c97c5c796-nxmnc", "messageParameters": {"p1":"failed to connect to `host=frarpo-r-0294-default- flexpsql.postgres.database.azure.com user=pgadmin database=SharedServices`: server error (FATAL: database \"SharedServices\" does not exist (SQLSTATE 3D000))"}, "messageKey":"failed to initialize database, got error%v", "properties":{"logger":"commons/storage/gorm", "file":"/go/pkg/mod/gopkgs.sas.com/sonder/v2@v2.42.2/storage/gorm/gorm.go:406", "caller":"gorm/gorm.go:607"},"timeStamp":"2024-06-26T09:43:46.992640+00:00","message":"failed to initialize database, got error %v"} … {"level":"error","version":1,"source":"sas-data-server-operator-c97c5c796-nxmnc","messageKey": "Reconciler error", "properties":{"error":"database server is external and cannot connect to the SAS database", "caller":"logr/logr.go:49"}, "attributes":{"controllerKind":"DataServer","DataServer":{"name":"sas-platform-postgres","namespace":"test"},"namespace":"test","name":"sas-platform-postgres", "reconcileID":"f7de99f8-f6b0-4a96-9e6f-53a9a8570ab5", "controller":"dataserver","controllerGroup":"webinfdsvr.sas.com"}, "timeStamp":"2024-06-26T09:43:46. 992706+00:00","message":"Reconciler error"}   An important detail…   This change, in the way the DSO handles an external PostgreSQL, relies on the presence of a specific annotation on the DataServer Custom Resource Definition which is only added if the new version of the PatchTransformer template delivered from version 2024.06 is applied.   When the DSO detects this annotation, it does NOT create the SharedServices Database in case it is not there. Instead, it only logs the fact that the SharedServices database is not there, with an "error" message (as shown above).   Here is an extract of the new PatchTransformer file with Stable 2024.06, usually called platform-postgres-dataserver-transformer.yaml or cds-postgres-dataserver-transformer.yaml (per the README instructions).     So if this sas.com/database-server-location annotation is NOT set for the DataServer Custom Resource, then there is no change and the SharedServices database is created by the DSO, in the external PostgreSQL server, just as before.   Here are some examples of cases in which the new annotation is not added:   As of today, if you deploy SAS Viya with the DaC tool - viya4-deployment GitHub project which has its own copy of the xxx-dataserver-transformer.yaml (without the annotation). or if you reference a static version of this configuration file in your Kustomization.yaml file from a previous SAS Viya version in your deployment of 2024.06.   In either of these situations, you won't see any change of behavior – the SharedServices database is automatically created, as before, by the DSO, even for an external PostgreSQL server.       Why?   The reasons leading to this DSO change in 2024.06 are related to security concerns :     Here is the specific requirement that could be problematic for security conscious database administrators:   The user that is the database owner must also have CONNECT privileges on the system database named postgres to enable SAS Viya platform services to start up.   The long-term goal is to remove the need to have such privileges associated to the PostgreSQL account used in the SAS Viya configuration.   The 2024.06 change removes the need for the Data Server Operator (DSO) to require this privilege (If a customer uses external Postgres and correctly pre-creates the SharedServices database, then the DSO won't connect to the "postgres" system database in the first place).   However…if you look at the external PostgreSQL requirement, you can see that we still have this requirement ☹:     It is because, outside of Data Server Operator (DSO), the Java services of the Viya platform still require a user with the CONNECT privileges to the system database (named postgres). Work is in progress to move to a newer Java framework and once all Java services will be using it, then there would be no need for a PostgreSQL account with to the system database for an external DB server.   With 2024.06, we are taking a first step toward the goal to reduce the permissions requirements on the Postgres user account used by Viya to interact with the external PostgreSQL server.     How?   Now…How can we avoid our deployment with the external postgres failing when we deploy stable 2024.06 ?   Pre-create the SharedServices Database   From the 2024.06 version of SAS Viya, if you are using external Postgres then you need to create the SharedServices database before starting the deployment process. Some guidelines are provided in the official documentation for this new requirement:     But while there is a link to the official PostgreSQL documentation, there are no command examples.   To create the SharedServices database, you should be able to run the code below with the user that you were passing into DSO.   CREATE DATABASE "SharedServices" WITH TEMPLATE template0 LC_COLLATE 'C' LC_CTYPE 'C'; REVOKE ALL on DATABASE "SharedServices" FROM public;   This is the SQL command that the DSO was running until 2024.06 (and keeps running when the PostgreSQL instance is internal).   That seems pretty easy, right?   Yes it is…As long as you are connected to the PostgreSQL database with an account with enough privileges to do so.   In customers environments, the database administrator should be able to connect to the external PostgreSQL server and run this command with the appropriate account.   Otherwise, here is a little trick to do it with a few commands.   Pro-tip: Create the Shared Service from an ad-hoc pod   The idea is to start an ad-hoc temporary pod, in the Kubernetes cluster, to run a psql command to create the Database.   In the example below, we set our information in environment variable and then we instantiate a pod in which we can run the command to create the database.   # Set required postgres variables PG_ADMIN_USER=pgadmin POSTGRES_PASSWORD="<my-super-password>" PG_SERVER=frarpo-r-0294-default-flexpsql.postgres.database.azure.com PG_PORT=5432 NS="test" # Create the SharedServices database kubectl run postgresql-dev-client --rm --tty -i --restart='Never' --namespace $NS --image docker.io/bitnami/postgresql:13 \  --env="PGPASSWORD=$POSTGRES_PASSWORD" \ --command -- psql -U $PG_ADMIN_USER --host $PG_SERVER  -d postgres -p $PG_PORT   This command opens a psql prompt:     Then you can type the SQL commands noted above to create and secure the SharedServices database:     The SharedServices database should be created before deploying SAS Viya.   However, if this requirement has been forgotten, it is still possible to do it even after the SAS Viya deployment has failed (due to the missing SharedServices database). But you must delete the DSO pod (sas-data-server-operator), after you run the SQL command.  When a new DSO pod is started, it detects the SharedServices database and, thanks to the Kubernetes auto-healing capabilities, all of the SAS Viya pods should recover to a running/ready state (this scenario was tested in our GEL lab).   Specific case: If you use the IaC tool for Upstream opensource Kubernetes   Finally, there is another particular case where you would NOT be impacted by this change.   If you are deploying SAS Viya in upstream open-source Kubernetes and if you decide to use the "SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes" to automatically install the PostgreSQL server, there is already an ansible task that creates the SharedServices database and grant all the privileges to the configured PostgreSQL administrator account.     In this case, the DSO change with 2024.06 will be transparent for you.   Conclusion   There is always a tradeoff to make between simplicity and security.   This change adds a requirement, with an additional task (pre-create the SharedServices database), either for the customer’s database administrator or for the person in charge of the Viya deployment. Despite the additional work, the expectation is that the reliance on a PostgreSQL super user account (with permissions on the postgres system database) can be completely removed in later SAS Viya versions.   That would allow SAS Viya to support the Zero Trust principle of "least privileges".       Find more articles from SAS Global Enablement and Learning here. ... View more

My colleague, Edoardo Riva explained last year, in an excellent blog post, how to use a local storage provisioner in Openshift. A similar technique can be used in an Upstream Open-Source Kubernetes cluster, to expose local disks attached to the Kubernetes nodes and leverage the available storage for SAS Viya volumes.   This is especially useful for two specific SAS Viya components that are installed as part of the Viya deployment and require block storage: Redis and OpenSearch.   As this post became longer than initially intended, I’m adding a little "ToC" below to help you navigate through its sections 😊    A gentle reminder on the Viya dynamically provisioned volumes Why do I need local block storage ? Local storage requirements in Upstream Open-Source Kubernetes Could you show me how it works ? Mimic directly attached disks in RACE See the local volumes Configure Redis and OpenSearch Can I switch from NFS-based to local-storage in an existing environment ? Conclusion   A gentle reminder on the Viya dynamically provisioned volumes   The SAS Viya platform requires various types of persistent data to function. During the deployment of SAS Viya, a lot of distinct volumes are automatically and dynamically provisioned and made available to Kubernetes as "Persistent Volumes" for the various Viya components (Consul, RabbitMQ, CAS, backups, etc…)   As an example, the table below lists the typical Persistent volume claims with the capacity claim, and required access modes by category.     When you deploy SAS Viya, if you haven’t set specific instructions (as noted there) in your   kustomization.yaml file to point to a specific storage class for some of your volumes, then the "default" storage class is used for ALL these persistent volumes.   This "default" storage class, that Kubernetes refers to, is configurable. It likely could be pointing to a local block storage implementation (usually the case in Public Clouds) or specifying an NFS-based storage (for example provided by the nfs subdir external provisioner).   Here is an example of how it looks like, if you have been using the IaC and DaC tools to deploy SAS Viya in an Upstream Open-source Kubernetes cluster :     In this case, the "default" storage class is provisioned by the nfs-subdir-external-provisioner-default which corresponds to exported directories from an NFS server.   While it may work (NFS can be used for both RWX and RWO access type volumes), it is usually not a good idea to use a single "default" storage class for all your volumes.   Why do I need local block storage ?   While NFS-based storage is acceptable for several of the volumes that are automatically created as part of the SAS Viya deployment, it is not adequate for Redis and OpenSearch components.   It is respectively noted in the SAS documentation here:     and there:     as well as in the official Redis configuration recommendations     and OpenSearch documentation:     So, as written everywhere, using NFS-based storage for Redis and OpenSearch in production system is not recommended due to potential locking and performance issues.   In addition, at least two recent "file access" issues have been observed in the field with this message in the Redis pod's logs :  "Fatal: can't update cluster config file." The problem was encountered  during a Viya version update in AKS with Redis using Azure Files and was solved by setting up an additional (block) storage class for Redis.   While not reported in our Technical support systems, the same error was also encountered during the update of the Viya version in an Upstream Open-Source Kubernetes platform at a customer in Germany during a deployment.   These "file locking" issues confirm the need to avoid NFS-based storage for Redis.   OK so, now that you are convinced, let's see how to get local block storage for Redis and OpenSearch in Upstream Open-Source Kubernetes !   Local storage requirements in Upstream Open-Source Kubernetes   While not explicitly mentioned in the "Platform-specific Requirements" page, since the Redis and OpenSearch volumes require block storage, it naturally becomes a requirement to provide directly-attached block storage for any Kubernetes cluster.   When SAS Viya is deployed in Public Clouds it is much easier to fulfill this requirement, the default storage class is typically adequate for Redis and OpenSearch (Azure Disks, GP2, GCE PD, etc…). But, when SAS Viya is deployed on-premise, it can be more complicated : Each Kubernetes node, where the Redis or OpenSearch pods could be scheduled, should have directly-attached volumes that could be used for the objects that these components need to persist. It is something that usually requires planning and adds up in the infrastructure costs (in the on-premise infrastructures, local disks or Enterprise grade SAN are typically configured for block storage).   As an example, I was recently contacted by colleagues (from SAS Germany) working on a customer deployment of SAS Viya in Open-Source Kubernetes where this requirement had not been identified in advance. It now results in the urgent need to provision and attach disks to the Kubernetes nodes to meet our official requirements ☹   The IaC for Kubernetes requirements page has a storage section to spell out what is needed and how the local-storage storage class can be used for specific Viya volumes:     If this requirement is fulfilled (disks with empty partitions attached to the Kubernetes nodes), the IaC tool will find the disks, format and mount them on the nodes, so the local storage provisioner can make them available through a storage class.   However, remember that the customers clusters are not necessarily created with the IaC Tool…So these attached disks on the Kubernetes nodes, and the way to make them available as block storage to Kubernetes, is something that should always been discussed during your Architecture design phase if you are deploying SAS Viya on-premise.   Could you show me how it works ?   Yes, I can show you how it works and if you have a subscription to the SAS Viya Deployment workshop in learn.sas.com you can even practice 😊   However, in order to build this hands-on where we can use the local storage provisioner and a local-storage storage class in our lab environment (based on VMware hosts) I had to use a special trick…   Mimic directly attached disks in RACE   In a real customer situation you would either fulfill the IaC requirement (disks with empty partitions attached to the Kubernetes nodes) and let the IaC tool format and mount the disks or mount your own Filesystems (from local disks or SAN storage) on the hosts.   Since we don't have the options to get such disks layout on our RACE machines, we need to simulate "fake" mounted disks.   Based on an example provided in the sig-storage-local-static-provisioner project we create several volumes directories (under /mnt/sas/volumes and mount tmpfs filesystems for each of them.   mkdir -p /mnt/sas/volumes for vol in vol1 vol2 vol3; do mkdir -p /mnt/sas/volumes/\$vol sudo mount -t tmpfs  -o size=15G,nr_inodes=15k \$vol /mnt/sas/volumes/\$vol done   The tmpfs filesystem (extension of ramfs) is a File System that keeps all its files in virtual memory and is temporary (in the sense that no files are created on the hard drive).   The script above is for one node, but you can easily place it in a file, then use ansible to distribute and run it across the kubernetes nodes that require local storage.   Important : Keep in mind that what we do here in these lab environments to simulate real local disks is not recommended in a real production environment.   See the local volumes   What is important to understand is that : the local storage provisioner installed by the IaC tool (or that you can install by yourself if the cluster was installed by the customer) is using Kubernetes DaemonSets, running on the nodes, which are constantly looking for mount points that could be used as local volumes.   So as soon as the new Filesystems appear on the Kubernetes nodes, they are automatically detected by the local storage provisioner pods running on the nodes, and if you check your persistent volumes, you should now see something like the following :     We have 9 available volumes that correspond to our vol1 , vol2 and vol3 mount points on 3 nodes.   Note that the local-storage storage class is always installed by the IaC tool for open-source Kubernetes with the following specifications:     A key part of the equation here is the  WaitForFirstConsumer "VolumeBindingMode".   It means that the volumes won't be allocated until the first pods request them and bind them through a PersistentVolumeClaim. The volumes, locally available on a given node, are taken by the first pods scheduled and starting on that particular node. It has an impact in the planning/provisioning of the volumes depending on the expected pods placement.   Configure Redis and OpenSearch   Now, the last thing to do is to make Redis and OpenSearch point to our local storage (instead of using the default storage class) and validate that the Redis and OpenSearch volumes are bound to our local PVs !   Here are the Kustomize Patch Transformers that I have been using to force OpenSearch to use the local-storage Storage Class:   tee ~/project/deploy/gelenv-localstorage/site-config/localvolumes/opendistro-storage.yaml > /dev/null << EOF --- apiVersion: builtin kind: PatchTransformer metadata: name: sas-opendistro-storage-class-transformer patch: |- - op: replace path: /spec/defaultStorageClassName value: local-storage target: kind: OpenDistroCluster --- apiVersion: builtin kind: PatchTransformer metadata: name: sas-opendistro-volumesize-transformer patch: |- - op: replace path: /spec/nodes value: - name: default replicas: 1 roles: - master - data volume: resources: requests: storage: 12Gi heapsize: 2G target: kind: OpenDistroCluster EOF   Note that in the second part of the PatchTransformer we are also changing the requested storage size for OpenSearch from the default value (128GB) to 12GB. If you use real local storage, you should not change the default value, but if you are just testing with memory-based filesystem (as I do in the Hands-on), then it might be useful.   For Redis, we simply need to change the storage class name :   tee ~/project/deploy/gelenv-localstorage/site-config/localvolumes/redis-storage.yaml > /dev/null << EOF --- apiVersion: builtin kind: PatchTransformer metadata: name: redis-modify-storage patch: |- - op: add path: /spec/storage value: size: 1Gi class: local-storage target: kind: DistributedRedisCluster name: sas-redis-server version: v1alpha1 group: redis.kun EOF   But by default, there are 6 Redis instances in a Viya deployment, so you need to make sure that you have enough available local volumes on the nodes where the Redis pods will be scheduled. Finding a local PV now become an additional constraint for the Kubernetes scheduler…if it can't find a suitable node with an available local volume, then the Redis pod cannot start.   At the end, after having referenced the PatchTransformer files from you main kustomization.yaml file, built your manifest and applied it (assuming you are deploying with the manual method), the OpenSearch and Redis pods should be ready for service and it should look like this in your Persistent Volumes view :     Note that the patch Transformer examples to change the OpenSearch and Redis default storage class are documented and available in the README files.    Can I switch from NFS-based to local-storage in an existing environment ?   Now, you might have already deployed your environment using unsupported NFS-based volumes for OpenSearch and Redis, and now you (or your customer) realize that we are not following the official SAS requirements. So, you might wonder : "can I change it without breaking my environment ?"   The answer is yes for Redis and OpenSearch (unlike other Viya components) because their internal databases can be considered as ephemeral storage since they are rebuilt every time these services start.   However, it requires an outage. You have to stop the SAS Viya platform deployment (using the sas-stop-all cronjob), remove the existing OpenSearch and Redis volumes, remove the OpenDistroCluster and DistributedRedisCluster instances. Then you can apply your Kustomize changes, and then restart all the SAS Viya platform deployment (using the sas-start-all cronjob).   If you want to experiment, here is the link to the SAS Deployment workshop where we do all these tasks.   Conclusion   I hope you have found this post useful. The main takeaway here, is that it should not be forgotten (during the Architecture design phases) that local block storage is a requirement for some Viya platform components (so it should be made available in your Kubernetes cluster).   Each Kubernetes platform (in the Cloud or On-prem) offer various storage options but they should be reviewed/tested in order to establish the Viya storage specifications : for all the volumes created during the Viya deployment but also for the data that need to be accessed from SAS Viya during the Analytics lifecycle.   While we have shown how to setup and use the local storage for Redis and Opendistro in this post, note that it could also be used for other things like the CAS Disk cache (instead of the hostPath volume type).     Find more articles from SAS Global Enablement and Learning here. ... View more

When SAS Viya 2024.02 was released, it added the support of Kubernetes 1.28 for all supported providers.   However, as noted in the "What’s new" page, Kubernetes 1.28 has implemented a change in behavior that could affect the CAS server if your cluster is configured with Linux cgroups v2.   In this post, after explaining how Kubernetes use cgroups to better control the cluster's workload and why this Kubernetes change is likely to impact the CAS server availability, we’ll discuss situations/use-cases where this problem could occur and review the possible workarounds.   But if you just want the TL;DR version, just jump to the conclusion paragraph 🙂   The issue   Cgroups (also known as "control groups") on Linux provides a way to better control resources such as CPU time, memory among processes running on the system.   Kubernetes relies on cgroups (either v1 or v2) to enforce the containers limits for CPU and memory which are defined inside the pods “resources” sections.   Here is how it works:   if a container, running inside the pod, uses more CPU than the defined limit, then it is throttled and if a container uses more Memory than the defined limit, then the Linux "Out of Memory" (OOM) Killer terminates the process inside the pod.   Sometimes, a picture is worth a thousand words, so here is a nice one for the cgroups that was posted on X (formerly known as twitter).   (source: https://twitter.com/b0rk/status/1214341831049252870)    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In our SAS Viya Deployment manifests, we have memory and CPU limits in the resource definitions to ensure that the Viya pods (including CAS pods) are not using more CPU or memory than expected.   For example in the CAS container : if one of the process (that corresponds to a CAS session), starts to use more memory than defined, then the OOM Killer will kill the process and the CAS session is terminated to preserve the CAS pod and the underlying node.   But Kubernetes, in its version 1.28, introduces a significant change in the cgroups v2 implementation,   If using cgroups v2, then the "cgroup aware OOM killer" is enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup.   If I had to represent this new "cgroup aware OOM killer" our little cartoon, it would look like that :   (source: https://twitter.com/b0rk/status/1214341831049252870)   From Kubernetes 1.28, if cgroups v2 are enabled then : instead of killing individual CAS sessions process, when the sas-cas-server container memory limit is exceeded, this new "cgroup aware OOM killer" now kills ALL the processes which are part of the CAS cgroup (which means all the other CAS process running on the node, so the whole Controller or Worker pod would be terminated).   Am I affected by this issue?   You might already have faced this CAS OOM problem in the field. It is a known issue which is already documented in the official "CAS Troubleshooting" section of the official documentation.   Here is an extract:   Explanation: This error message indicates that a session was killed. One possible cause is that the out-of-memory (OOM) killer terminated a session because of high memory use.   But while users could just restart their CAS session and try again in case the problem arise, this event could be much more disruptive in Kubernetes 1.28 (because of the memory.oom.group change)…   According to the SAS Technical Support experience, some CAS actions started by Model Studio and Risk solutions, for example, are frequent CAS killers due to an intensive usage of memory. When the memory usage is not backed by enough space in the CAS_DISK_CACHE, then one session could exceed the CAS container’s configured memory limit and trigger the OOM Killer. If your environment is already under pressure of CAS sessions and users, you can check in your log if these OOM have appeared.   Here is what you’ll see in the sas-cas-server container log, in case you hit the CAS OOM issue :   Child terminated by signal: PID nnn, signal 9, status 0x00000009   You can also check the CAS node(s) system log and search for messages like :   Feb 14 16:59:38 sasnode06 kernel: [11768.075646] Memory cgroup out of memory: Killed process 525176 (cas) total-vm:6054896kB, anon-rss:3983660kB, file-rss:60768kB, shmem-rss:4kB, UID:1001 pgtables:10980kB oom_score_adj:1000   If you see these kind of messages in your logs and are using cgroups v2, then you are very likely impacted by the change coming with Kubernetes 1.28.   If you are already facing this OOM issues today in your environment, one option is to bump the resources up on the CAS nodes to resolve these OOM events but other workarounds are provided below.   So what can I do ?   Limit the risk of having CAS sessions OOM Killed   The OOM Killer is triggered as soon as the sum of memory used by the CAS process on the node exceeds what has been set in the sas-cas-server’s memory limit ("resources" section in the pod's definition).   While before Kubernetes 1.28, it would affect only one CAS session and leave the others alive, it will now kills the whole CAS system (if CAS is SMP) or the whole CAS worker pod (for MPP CAS), bringing down all the CAS sessions running there.   So one way to significantly reduce the chances of experiencing the problem, is to run CAS with sufficient memory allocation (whether it is with a Guaranteed Quality of Service or with the CAS auto-resources configuration) and to provision additional capacity in case the OOM issue appear.   For example, with the default CAS auto-resources configuration (as explained in this post), the sas-cas-server container's memory limit is set to the amount of available RAM on the underlying CAS node.   With the CAS auto-resources configuration, if one or more process/CAS sessions causes the OOM, it means they use more memory than the available RAM on the machine, which it is a good indicator that the infrastructure is not adequate for the workload… Even without the specific "cgroup aware OOM killer" issue, in such a situation the whole CAS instance might be killed as Kubernetes would evict the CAS pod to prevent the node’s resources exhaustion.   Stay on Kubernetes 1.27, until a fix is available    A PR (Pull Request) has been opened in the Kubernetes GitHub repository to add a "singleProcessOOMKill" flag to the kubelet configuration. Setting that to “true” enable single process OOM killing in cgroups v2. In this mode, if a single process is OOM killed within a container, the remaining processes will not be OOM killed."   These changes are currently targeted against Kubernetes 1.30. So another option would be to stay on Kubernetes 1.27 and wait for a newer Kubernetes version that provides a way to disable the "cgroup aware OOM killer". According to the current planning, the support of Kubernetes 1.30 could be added in September with SAS Viya 2024.09 (we would drop 1.27 at the same time).   However, it might not be easy to change the kubelet configuration (especially in Cloud Managed Kubernetes environments) and this new feature remains to be confirmed.   Another problem is that jumping from Kubernetes 1.27 to 1.30 is a major change and may not be aligned with the customer upgrade strategy for their Kubernetes environment (except maybe in the case where they opted for the LTS tiers, if made available with K8s 1.27 by their cloud provider).   Use cgroups v1 for CAS node pools   Changing the cgroups type might be a better option.   Remember that this problem only affects CAS nodes configured to use cgoupsv2. So if you are using workload placement for your CAS pods, ensuring that the nodes in your CAS node pool use cgroups v1 would be a way to avoid this issue.   If you are not sure which cgroups implementation is used in your environment, here is a command that you can run, on your CAS nodes, to determine if cgroups v1 or cgroups v2 is used:   $ stat -c %T -f /sys/fs/cgroup   Example output for cgroups v1 implementation:   tmp2fs   Example output for cgroups v2 implementation:   cgroup2fs   While most Cloud managed Kubernetes service are now using cgroups v2 by default, it is still possible to switch back to cgroups v1.   The table below summarize cgroups implementation and ways to switch to cgroups v1 for the Kubernetes Distributions currently supported for a SAS Viya deployment (the information here comes from the R&D work and efforts on this topic).   Kubernetes implementation Notes AKS (Azure) A Daemonset can be used to fall back to cgroups v1, see this page for details and instructions EKS (AWS) According to this page, at present, EKS does not offer an optimized AMI that supports cgroups v2. Its availability is currently being tracked in this GitHub issue.   However it remains possible to use other images (AMI) to configure EKS with cgroups v2. GKE (GCP) When the GKE cluster is provisioned one can select between cgroups v1 and cgroups v2 in node configurations. The default is cgroups v2. OCP (RedHat Openshift) As of OpenShift Container Platform 4.14, OpenShift Container Platform uses Linux control group version 2 (cgroups v2). But it is also possible to configure cgroups v1 in openshift. Upstream Open Source Kubernetes (OSS) Cgroups are discussed in the Kubernetes project documentation. The kubelet and the underlying container runtime rely on the version of cgroups that is enabled on the underlying linux nodes (see above to determine which cgroups version is used in your own case.)   Conclusion   One of my SAS R&D colleagues gave a very good summary of the issue that is discussed in this blog post:   “On Kubernetes 1.27, the OOM killer would just kill the PID that had the worst OOM score.  Which usually/hopefully, would just be a CAS Session that is hogging a lot of memory.  Therefore just that session would die, but the rest of the users would be unaffected. In 1.28, it kills all the PIDS which take down the whole CAS server.”   While this change in Kubernetes presents a real risk for customers with a significant CAS workload, the good news, though, is that this specific issue has been detected quite early, researched, reviewed, communicated, and well documented. A great job has been done by the SAS R&D as a whole, to provide workarounds and guidance to customers if the issue seen in the field.   In a companion and follow-up post, I will get my hands dirty and show how to reproduce this issue with a simple SAS program and highlight the differences between an environment that is affected by this change versus an environment that is not. I will also show how to implement the cgroups v1 change workaround.     Find more articles from SAS Global Enablement and Learning here. ... View more

When the SAS Deployment Operator was initially released, one of its key capabilities was to read the SAS Viya configuration from a remote source (such as a source code repository or a Web server).   Today, the latest deployment method, known as the sas-orchestration command also offers this option.   This article discusses the benefits to read the SAS Viya Configuration from a remote git repository (instead of using local files) and explains how to do it with the sas-orchestration command. It also shows an implementation example.   Why?   A SAS Viya deployment configuration is comprised of two main items:   a kustomization.yaml "main" file, and the site-config folder files which are typically stored under the same file system structure on your jump host.   Managing these configuration items can become a bit challenging overtime and make it difficult, to know what has changed and when…   For traceability purposes you could, for example, create a new copy of your SAS Viya configuration folder for each update (each month if you are on a stable cadence, and each time you modify the configuration).   After a little while that historical group of directories might look something like:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   As you can see it is not very efficient; a lot of files are duplicated, and it could quickly become pretty messy…(and I’m not even showing the structure under site-config in the screenshot).   In addition, keeping the SAS Viya configuration on a jump machine’s file system means that you need to keep this machine around, make sure it is secured and backed up, etc… which is not very convenient or aligned with the current DevOps standards (where everything should be ephemeral and automatically built by a pipeline).   On the other hand, having your SAS Viya configuration files stored in a modern source code repository system (such as GitLab or GitHub) brings all the benefits of a modern code versioning system. You could organize your environments in different branches and keep track of all the changes, tag your deployed version and easily come back to them, etc…   It would also be much easier to integrate the SAS Viya configuration in a modern DevOps system.   Deploy SAS Viya from a remote location   The go-getter project provides a library, used behind the scene by the DaaS* tools, to pull the required SAS Viya configuration from a remote location.   As explained in the go-getter project’s README, “go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input”. Out of the box it supports the following protocols : local files, Git, Mercurial, HTTP, Amazon S3, Google GCS.   So, in theory, as long as you can setup the authentication to the remote repository and figure out the right “go-getter” URL (in the Deployment operator CRD or in the sas-orchestration command line), you could place and manage your SAS Viya configuration in any storage system (providing access through one of these go-getter protocols).   However, the current reality is a bit different 😊   While go-getter is used by the DaaS tools, it does not mean every possible system and authentication combination will automatically work when used from the sas-orchestration tool or Deployment operator.   For example, local files and "Git over SSH" access is currently not supported by the DaaS tools. So far, the only "go-getter" URLs tested and supported are the ones using the "Git over http(s)" protocol, like :   For a public repository : git::http://github.com/jsmith/public.git//folder For a private repository: git::http://jsmith:<git_token>@github.com/jsmith/myviya.git//conf   Also, note that the "Git over http/s" go-getter URL only works for the SAS Viya configuration files ( kustomization.yaml and site-config folder files), but not for the order’s certificate information or the license file.   The DaaS tools can read the order’s certificate information or the license file either from a local file system or from an HTTP server.   (*) DaaS = Deployment as a Service: DaaS tools are the Deployment operator and sas-orchestration command. Using GitLab   In terms of infrastructure, we can install a containerized instance of the Open Source GitLab* platform inside our on-premise (Upstream Open source Kubernetes) or Cloud Managed Kubernetes Clusters (Google GKE, Amazon EKS or Microsoft AKS).   In such case, sometimes for demo or training puproses, it is possible to take some shortcuts, using the default "root" account in GitLab and making the project public, so we don’t have to perform any real authentication in the go-getter Git URL.   But a much better practice is to store and manage the SAS Viya configuration in a private GitLab repository.   So, in the next paragraph, we’ll show an example with the “SAS orchestration deploy command” using a git personal token to authenticate and to read the order’s asset and SAS Viya configuration from a private GitLab repository.   (*) GitLab is a very popular web-based Git repository that provides free open and private repositories, issue-following capabilities, and wikis. An example with GitLab and personal access tokens   Create your GitLab token   GitLab personal access tokens allow you to authenticate with Git using HTTP Basic Authentication. This is what is expected to be used in the sas-orchestration tool.   Users can easily create their own personal access token from the GitLab web interface.   All you have to do, once you have signed in, is to right click on your GitLab avatar image, select "Edit Profile", then click on "Access Token".     Then you simply give a name to the token, consider changing the expiration date, select a scope ( read_repository and write_repository scopes are sufficient) and click on the blue button "create personal access token".   Once it is done, you should see a message that confirms that "Your new personal access token has been created."     Now, it is important that you take note of the token content (you can reveal it or just copy it in your buffer), because we will need it later when we run the deployment command.   Store your SAS Viya configuration in GitLab   After that, you can create your "Viya" GitLab repository and store your SAS Viya configuration files there. It can be done either with the GitLab web UI or through Git commands.   If you use Git commands, you’ll first have to setup and configure the Git client on your machine, here is an example (on Linux):   git config --global credential.helper store git config --global http.sslVerify false git config --global user.name "GEL Student" git config --global user.username jsmith git config --global user.password "lnxsas123" git config --global user.email "GELStudent@mycompany.com" These commands update the Git configuration file located in the user’s home directory ( ~/.gitconfig ). You can easily check your Git configuration just by running " git config --list ".   In our scenario, we assume that we don't have the SAS Viya configuration in the remote repository yet.   So, we first need to create a new set of  SAS Viya configuration files ( kustomization.yaml and " site-config " directory) in a local folder.   Once they are in the local folder, we can initialize the repository, reference the remote location, perform our first commit (and finally push to our remote GitLab repository).   You can find below an example of the commands to do that (on a linux machine).   # move to the local repo folder (that contains our SAS Viya configuration files) cd ~/project/deploy/viyaorch-repo # Initialize the git repository git init # Reference the remote location (it also creates the project in GitLab). git remote add origin http://gitlab.devops.company.com/jsmith/viyaorch-repo.git # Now commit our changes in the local git repository. git add . git commit -m "Initial commit of the viyaorch configuration."   You would see something like that:     At this point the Git repository has been initiated but it remains local.   Now it is time to "push" our changes toward the remote GitLab repository.   git push http://gitlab.devops.company.com/jsmith/viyaorch-repo.git     The first time, you are asked for your git username and password*, but then your credentials (that are in your .gitconfig file) are stored  for the next time (remember the credentials.helper parameter was set to store ).   Back to the GitLab web interface, you should then be able to see your SAS Viya configuration files in your GitLab project’s repository:     Congratulations, your SAS Viya Configuration is now stored in GitLab ! 🙂       (*) If you want to avoid this first time prompt, you can create a ~/.git-credentials file in advance with the credential in the following format :  "http://<git username>:<git password@gitlab.devops.company.com" Deploy SAS Viya   The final step is to set up the go-getter URL in the sas-orchestration command and run it to deploy SAS Viya!   In the user-content parameter for the sas-orchestration command, we specify the GitLab personal access token that we created earlier (instead of the user’s GitLab password).   Here is an example:   docker run --rm \ -v $(pwd):/workingdir/ \ -v ~/.kube/config:/kube/config \ -e "KUBECONFIG=/kube/config" \ --user $(id -u):$(id -g) \ sas-orch \ deploy \ --namespace viyaorch \ --deployment-data /workingdir/viyaorch/SASViyaV4_certs.zip \ --license /workingdir/viyaorch/SASViyaV4_license.jwt \ --user-content "git::http://jsmith:<gitlab personal access token>@gitlab.devops.company.com/jsmith/viyaorch-repo.git//config" \ --cadence-name stable \ --cadence-version 2023.06   Note that in the example we still read the order’s certificate ( SASViyaV4_certs.zip ) and license ( SASViyaV4_license.jwt ) files from the local filesystem in this case (but they could also be read from a remote HTTP server).   Finally, if you would like to experiment by yourself, note that the SAS Global Enablement and Learning team is developing specific SAS® Viya 4 deployment hands-on content within workshop style content. Contact your local SAS Education team for more details if this is of interest to you.   Cloud Git Repositories alternatives   While it makes sense in an "on-premise" environment to run a local GitLab instance, for deployment in the Public Cloud the use of the Cloud Provider code repository solution could be a better and more consistent option in the long run.   Your customer might even have this as a requirement if the Cloud Provider code repository is part of their standards and is already used for other projects.   Azure, AWS and Google Cloud Platform offer “git compatible” repository services, such as  Azure Repos, AWS CodeCommit or Google Cloud Source Repositories.   As explained before, while go-getter can used by the DaaS tools (for the "git over http(s)" protocol), it does not mean every possible Cloud Provided Git Repository system and authentication combination will work automatically when used from the sas-orchestration tool or Deployment operator.   However, in the next article we will show how (with some tweaks and tricks) the Google Cloud Source Repositories service can be used as the SAS Viya configuration source for a SAS Viya deployment in the Google Cloud Platform.     Thanks for reading! ... View more

In the first part of the series, we focused on the reasons to choose an external PostgreSQL instance for the SAS Viya Data Infrastructure Server and how to provision it with the IaC tool for Kubernetes.   In this 2 nd article, we detail how the viya4-deployment GitHub project (aka DaC tool) can now be configured to deploy SAS Viya (with an external PostgreSQL server).   Then we'll open up on additional considerations around the utilization of an external PostgreSQL instance for the Viya platform (HA, connections, tuning , technical benefits).   Finally, in the last section of the article, we compare the Kubernetes cluster CPU and memory resource usage and reservation between two environments (one using the internal Crunchy Data Server and one using the external PostgreSQL instance running on a distinct VM).   How to configure SAS Viya for it (with the DaC tool)   The option to use an external PostgreSQL server as the SAS Infrastructure Data Server has been available for some time.   This excellent article from Edoardo Riva is the perfect starting point to understand how to configure Viya to use it.   Basically, you have to provide your Viya environment with some connection information and credentials, so all the Viya components that need to interact with the PostgreSQL Server database (write or read data from the SAS Infrastructure Data Server) can do so.   Here, we just look at an example in the case of an Opensource Kubernetes platform with the most natural companion to the IaC, the viya4-deployment tool (aka "DaC" for "Deployment As Code").   PostgreSQL Server details   The variable used to define the PostgreSQL details (internal or external) in the DaC tool is named V4_CFG_POSTGRES_SERVERS and is actually a "map" of objects.   The variables documentation page shows the various available options, such as the server FQDN and port, the database name, the size of the internal PostgreSQL and pgBackrest PVCs, etc…   Note that several SAS Viya platform offerings require a second internal Postgres instance referred to as SAS Common Data Store or CDS PostgreSQL (the CDS PostgreSQL can also be provisioned by the IaC tool, as noted in the first part of the series).   Here is an example for an external PostgreSQL server :   # External servers V4_CFG_POSTGRES_SERVERS: default: internal: false admin: pgadmin password: "password" fqdn: mydbserver.local server_port: 5432 ssl_enforcement_enabled: true database: SharedServices other_db: internal: false admin: pgadmin password: "password" fqdn: 10.10.10.10 server_port: 5432 ssl_enforcement_enabled: true database: OtherDB Trusted certificate for the PostgreSQL Server   In order to encrypt the communications between the Viya "clients" (all services needing to persist or request information in the SAS Infrastructure Data Server) and the PostgreSQL service we also need to provide the PostgreSQL certificate file location, so the DaC tool can add it in the SAS Viya trust store.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The PEM-encoded certificate file that we need is on the PostgreSQL server and, as shown in the first part of the article, its location can be found in the PostgreSQL settings.   SELECT Name,Setting from pg_settings WHERE category='Connections and Authentication / SSL';   So, in this case the V4_CFG_TLS_TRUSTED_CA_CERTS variable in the DaC ansible vars file should point to the file located on the external PostgreSQL server in /etc/ssl/certs/ssl-cert-sas-rext03-0038.pem .   External PostgreSQL Considerations   HA PostgreSQL   The SAS Infrastructure Data Server component is a critical component of the Viya platform… if, at some point, it fails or is not working properly, the entire Viya platform stop working as designed.   So, running it as a "stand-alone" service would introduce a major SPOF (Single Point Of Failure) in the Viya platform architecture.   One of the benefits of the internal PostgreSQL server implementation with the Crunchy Data PostgreSQL operator is that we automatically have a clustered installation of PostgreSQL (1 primary and 2 replicas) which makes the SAS Infrastructure Data Server component Highly-Available.   Now the question is How can we have the same High-Availability with external PostgreSQL ?   There are various possibilities.   The external PostgreSQL implementation also matters.   As of today, according to the official  documentation we support the following options for the external  Postgres implementation:   PostgreSQL (Open Source) Microsoft Azure Database for PostgreSQL - Flexible Server Amazon RDS for PostgreSQL GCP Cloud SQL for PostgreSQL If you decide to use one of the supported Cloud managed external PostgreSQL database, you need to check their current documentation. However, they generally propose options for High Availability (HA) configurations with automatic failover capabilities.   For example in Azure, with the PostgreSQL Flexible server, “When high availability is configured, flexible server automatically provisions and manages a standby. Write-ahead-logs (WAL) is streamed to the replica in synchronous mode using PostgreSQL streaming replication.”   An alternative is to use the Open Source version of PostgreSQL.   It can be installed on a standalone VM. And that’s exactly what is done when using the IaC for Kubernetes tool. But it would not provide any High Availability by default. In order to have HA, the PostgreSQL should be deployed in a cluster of at least 3 machines and be complemented with something like PGPool for the request load-balancing and a virtual IP address (this is similar  to the Viya 3.5 architecture). Implementing such kind of setup is not trivial.   Another possibility would be to rely on Kubernetes itself to provide this HA capabilities by deploying  PostgreSQL with an operator. The operator could be deployed in the same Kubernetes cluster as Viya.   Here is a list of PG Operators implementation that could potentially be used:   Crunchy Data Zalando CloudNativePG Percona Stackgres  (credits to @Carus Kyle for providing this list 😊)   While the experience and feedback on these options are still limited, they will likely allow the installation of a Highly-Available PostgreSQL server. It would be up to the customer team to ensure the ongoing operation of this PostgreSQL server. While these implementations are opensource, most of them (such as CloudNativePG or Percona) can come with a "full support" model provided by specialized vendors.   Pgbouncer   One of the challenges when using a Cloud Managed Service for the external PostgreSQL is to ensure that the maximum number of connections to the database is high enough for SAS Viya (we have a lot of microservices that need to open connections with the PostgreSQL database).   As stated in the official requirements : “An external PostgreSQL server should support a maximum number of connections, max_connections on some providers, and max_prepared_transactions of at least 1024.”   Cloud providers sometimes apply limits and thresholds depending on the instance type that underpins the database, including on “max_connections”. For example, it used to be problematic with the Azure Database for PostgreSQL - Single Server as higher (and very costly) pricing tier was required to meet this requirement.   Today though, the default Azure Database for PostgreSQL implementation is "Flexible Server" and it supports up to 5000 connections. It is much better, however even with 5000 connections, it might not be enough for  some  large multi-tenant environments.   That’s where PGBouncer can help. You might have heard about it, it is also briefly mentioned in our SAS Viya tuning guide.   PGBouncer is a connection pooler for PostgreSQL. It allows a larger number of frontend connections to share a smaller pool of backend connections. The aim of PGBouncer is to lower the performance impact of opening new connections to PostgreSQL, by re-using database connections.   Finally, note that Kubernetes Operators for Postgres, such as Zalando usually include the PGBouncer component by default.   Technical Benefits?   There are several benefits in the utilization of an external PostgreSQL implementation for the SAS Data Infrastructure server. Here is a screenshot of a colleague’s comment (in an internal discussion about pros and cons of the "external postgres" decision) that I found quite enlightening :     Finally, from my perspective I was also curious to know if, there was a significant reduction for the Kubernetes Hardware requirement when the SAS infrastructure Data server was located outside of the Kubernetes cluster? So it made me wonder “What is the impact on the cluster resources usage of running PostgreSQL outside of the Kubernetes cluster ?”   Well, actually it is the question that I have tried to answer in the next section 😊   Resource consumption comparison   When running the OOTB Crunchy Data Server in the Viya namespace, we have 3 PostgreSQL pods corresponding to our clustered instances as well as the Postgres Operator pod and a PgBackRest instance pod when backups are running.   So, surely replacing that with an external PostgreSQL outside of the cluster will reduce the overall Viya CPU and memory footprint on the cluster itself !?   Since I was very curious about that, I made some comparisons using k9s and some of the Grafana Dashboards coming with the  SAS Viya 4 Monitoring for Kubernetes.   I deployed the same Viya order in 2 Upstream Opensource Kubernetes environments : one was deployed with the internal Crunchy Data Server and the other was using an extra machine with an external PostgreSQL Stand-alone deployment (outside of the Kubernetes cluster).   Please, note that these are very simple, "one time test", and the values reported there were varying overtime. Also the comparisons are made when the system idle, as we are not generating user activity.   It is clearly not something as thorough and systematic as an official benchmark done by our Performance teams at SAS. However, I found the results quite insightful and I believe it gives a rough idea of the differences in terms of resource reservation and utilization.   Kubernetes objects, overall CPU and memory utilization   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server ("External PostgreSQL")     As we can see with the k9s "pulses" view, running PostgreSQL outside of the cluster reduces the number of Kubernetes objects (Pods, ReplicaSets, StatefulSets, PersistentVolumes,). In our case the overall impact of the CPU and memory used is low, however it is one data point for a single measure in an idle and empty environment.   Overall resources usage   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server (“External PostgreSQL”)     While the overall cluster differences in terms of usage for the CPU and memory are not really significant (around 1%), there is a slightly bigger difference in the resource requests commitments (49.9% vs 53.2% for the CPU). Less CPU and memory resource are "reserved" on the nodes when the external PostgreSQL Server is used.   CPU utilization for the Viya namespace   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server ("External PostgreSQL")     Now if we specifically zoom on the Viya namespace ("dac"), we clearly see the difference in the CPU Requests between the 2 environments (9.23% vs 6.57%).   MEMORY utilization for the Viya namespace   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server ("External PostgreSQL")     Finally, the two Grafana graphs confirms a relatively low difference on the memory usage in an idle/empty environment.   Here are the main takeaways of this first and quick comparison : The amount of CPU and memory resources reserved on the Kubernetes is reduced when PostgreSQL is running outside of the Kubernetes cluster (each of the four Crunchy Data PostgreSQL Instance requests almost half of a CPU and 3GB of memory), ...but it remains a relatively small reduction in the overall Viya Platform resources requirements. In an idle, freshly deployed environment without any user’s activity, there is not significant difference in the CPU and memory utilization between the two environments. Conclusion   And that’s all for this time.   I hope this  series was useful to reiterate some of the established architecture principles and configurations for the SAS infrastructure Data Server component … Thanks for reading ! ... View more

A few weeks ago, I was contacted by my colleague Frederik (from the Global Technology Practice) as his large UK customer was challenging him on our requirements, in terms of permissions in Kubernetes for the deployment and administration of SAS Viya.   It made me realize that when we deploy SAS Viya internally at SAS or in our Deployment or Administration lab environments, we often do it with the Kubernetes administrator full access. With this kind of access, we can create but also delete any cluster-wide resource, such as someone else’s namespace, someone else persistent volume (where critical information might be stored), etc.   While it is very handy and make our life easier each time we need to deploy/redeploy or change anything in SAS Viya, it is quite unlikely the Kubernetes administrator, working for your customer, accepts to give you such kind of access (especially if the Kubernetes cluster is shared with other applications).   Instead, they will probably tell you: "Okay SAS admin, now could you tell me exactly what kind of RBAC* authorizations you need in my Kubernetes cluster to deploy your application?"   If you have experienced this situation or think you may face it sooner or later, you might find this blog useful. Its goal is to propose an approach where the Kubernetes administrator can delegate the administration of the SAS Viya environment to someone with limited privileges in the cluster.   (*) RBAC stands for Role Based Access Control, you can learn more about it there   Review of the existing resources   Viya ARK   The Viya ARK pre-installation playbook checks Creation/Deletion of: CRD, Role, RoleBinding, Deploy, Service, Ingress/Gateway objects.   For example, here is how the "Viya ARK Pre-Install check" report looks like when using a restricted service account (who is a namespace admin and has some additional cluster-wide read-only permissions on namespaces and PVs).   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The report is consistent, since to perform the whole deployment, Cluster admin permissions are required. However, it does not provide answers to the customer questions discussed in the introduction of this post.   Official Documentation   In February 2022, a new section “Cluster Resources and Roles That Require Elevated Permissions” was added in the SAS Operations documentation to provide additional guidance on this topic.   While the doc section remains at a high level (and could be confusing to someone who is new to the Kubernetes RBAC model), it explains that the Kubernetes cluster administrator could create additional roles for delegated administrators who could then deploy and administrate the software.   One of the recommendations is to provide “a delegated, namespace-level administrator with the "get", "list," and "watch" permissions on all resources cluster-wide.”   With this level of permissions, the delegated administrator (let’s call him the SAS Viya administrator) can see all resources (including secrets) in every namespace of the cluster. It is not something that will always be acceptable for customers…and as we’ll see with practical examples, it is not really required.   Challenge and suggested approach   When the SAS consultant cannot have the cluster administration privileges to perform the SAS Viya deployment and administration, the following approach could be followed to address the customer’s requirement.   Use the manual deployment method. Create a "SAS Viya admin" service account that will be used by the SAS consultant. Give permissions to the "SAS Viya admin" service account in the SAS Viya namespace. Split the initial manual deployment commands (depending on their "scope") between the Kubernetes Administrator and the SAS consultant. Only if needed, add specific permissions as needed for additional administration tasks.   What if I want to use an orchestrated deployment method ?   Important : Splitting the initial deployment tasks between the K8s admin and the SAS Consultant (for example with a namespace scoped privileges) is only possible with the manual deployment method.   If the chosen deployment method is the sas-orchestration, then the "cluster level" privileges are required each time the software has to be redeployed…which is something that happens each time you need to update your kustomization.yaml file and force SAS Viya to pick up configuration changes.   If the chosen deployment method is the SAS Viya Platform Deployment Operator, then the re-deployment is always managed through a reconciliation process that is handled by the deployment operator and run under the Deployment Operator's Cluster-role.   Scopes and sas.com/admin labels   The SAS Viya platform is made up of various Kubernetes objects.   Some objects have a “cluster-wide” scope (they are global for the whole cluster and not limited to a specific namespace), others have a "namespace" scope.   In the SAS Viya manifest (site.yml), each object is in a category (or scope) determined by a "sas.com/admin" label.   There are 4 categories (listed in the table below that comes from the SAS official documentation). However, these categories don’t necessarily match what Kubernetes considers as "namespaced" object (Configmaps, Pods, PodTemplates, Services, etc..) and "non-namespaced" objects (Nodes, PersistentVolumes, ClusterRoles, clusterRolebinding, etc…). As an example, apart from one exception, the "cluster-local" category is associated to "namespaced" objects.     The selectors "sas.com/admin=cluster-wide" and "sas.com/admin=cluster-api" are associated to Cluster level objects.   It means that the kubectl apply command, that target these kinds of objects, can only be executed by someone with the default  cluster-admin ClusterRole bound with a ClusterRoleBinding (which gives privileges at the cluster level – i.e. your almighty Kubernetes Administrator).   The selector "sas.com/admin=namespace" is associated with "namespace level" objects. So, someone with a namespace administrator role for the specific namespace, could run the kubectl apply command that targets these objects.   Then we have the "sas.com/admin=cluster-local" level, which is a mixed bag of various kinds of objects that also exist in the other categories. Since there is generally at least one object with this label that is at the cluster-wide level, a namespace administrator role would NOT be able to run the  kubectl apply command that target the "cluster-local" scope with full success…   The issue with the "cluster-local" label   This selector is a bit more problematic since it contains 99% of namespace level object ("configmaps", "secrets", "persistentvolumeclaims", "rolebindings", "podtemplates") but also potentially cluster-wide objects (ClusterRoleBinding for the SAS Workload Orchestrator).   Actually, in our test environment (with the "SAS® Viya®" offering), the only object that prevented us successfully applying the "sas.com/admin=cluster-local" level as a namespace administrator is the "sas-workload-orchestrator" ClusterRoleBinding object.   The issue with this "cluster-local" label is that : if for a given Viya configuration/customization, we need to modify an object that is at this level, then it will be necessary to run the kubectl apply command at the "cluster-local" level…So our namespace level Viya administrator will have to call the Kubernetes admin just to run the kubectl apply at this level (even if the modified object is at the namespace level ! ) ☹️   Removing the "cluster-local" label and assigning the associated resources either to "cluster-wide" or "namespace" categories, would greatly simplify things as it would allow Viya administrators with namespace privileges to perform almost any kind of Viya configuration/customization.   Required role and additional permissions   A recommendation of the official SAS Documentation is to provide "a delegated, namespace-level administrator with the "get", "list," and "watch" permissions on all resources cluster-wide."   Ideally, the namespace administration privileges would be enough (without the need to give permission on all resources cluster-wide). What we can do is to use one of the Kubernetes existing "built-in" roles.   As noted in the official Kubernetes RBAC documentation,  the "cluster-admin" built-in ClusterRole "when used in a RoleBinding, gives full control over every resource in the role binding's namespace, including the namespace itself."   Here is an example on how to give this role to a "viya-admin-lab" service account.   # create the manifest tee /tmp/create-binding-for-viya-admin-lab.yaml > /dev/null << EOF --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: sas-rolebinding-namespace namespace: lab subjects: - kind: ServiceAccount name: viya-admin-lab namespace: lab roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io EOF # apply the manifest kubectl apply -f /tmp/create-binding-for-viya-admin-lab.yaml   We simply create a "RoleBinding" object to attach the built-in "cluster-admin" ClusterRole to our "lab" namespace. So, while it is a "ClusterRole", it can be limited to the scope of a particular namespace.   With the above privilege definition, someone using our "viya-admin-lab" service account could change everything inside our Viya namespace but could not do anything else outside of the namespace.   Note: Here we assume that we have already created the "viya-admin-lab" service account and generated the associate KUBECONFIG file.   Break down the responsibilities   The table below shows the breakdown of tasks between the Kubernetes administrator and our SAS Viya administrator using the service account described above (with his privileges limited to the SAS Viya namespace).   Task Responsibility Notes Deploy SAS Viya Shared The Kubernetes administrator runs the "kubectl apply" commands for "cluster-api", "cluster-wide" and "cluster-local" levels. The SAS Viya administrator runs "kubectl apply" command for the namespace level (and potentially for the "cluster-local" level). Update SAS Viya Shared Stop/Start the SAS Viya platform SAS Viya administrator (namespace admin) This task is handled by Cronjobs defined at the namespace level. Backup and restore the SAS Viya environment SAS Viya administrator (namespace admin) Perform common customizations (change CAS or SAS programming runtime settings, change CAS topology, etc…) SAS Viya administrator (namespace admin) While the SAS Viya administrator cannot change the Custom Resource Definitions (CASDeployment definition), he can modify Custom Resource instances (CASDeployment instances), which is enough to change the configuration of a deployed CAS Server. Allocate additional permissions to the SAS Viya administrators when/if needed for specific configuration. Kubernetes Administrator We haven’t tested all possible configuration/customization. Some may exceptionally require additional privileges. The additional permissions would be added as required.     Note: The described tasks have all been tested in the GEL Deploying SAS Viya workshop Hands-on.     Conclusion   Hopefully this post helped you to understand which kind of authorization is required in Kubernetes for a SAS Consultant or a SAS Administrator to accomplish his tasks related to the SAS Viya deployments.   The content presented here comes from practical experimentation in a Viya LTS 2023.10 environment deployed on an Upstream Opensource Kubernetes cluster (in version 1.27). If you want to see these concepts by yourself in practice, note that the SAS Global Enablement and Learning team is developing specific SAS® Viya 4 deployment hands-on content within workshop style content. Contact your local SAS Education team for more details if this is of interest to you.,.   Bonus: Some useful commands to check permissions and scope   Finally, as a little bonus, you can find below some useful commands to explore the Kubernetes RBAC implementation.   "kubectl-whoami" and "kubectl auth can-i" commands. # the whoami plugin can be installed with krew () kubectl-krew install whoami kubectl whoami # can I create pods ? kubectl auth can-i create pod -n lab # can I create the resources corresponding to the "sas.com/admin=namespace" label in the SAS Viya manifest ? for res in service deployment replicaset replicationcontroller job cronJob statefulset daemonset podtemplate do # test the create privilege for each resource echo "testing creation right on $res" kubectl auth can-i create $res -n lab done   See Rob Collum’s post for more about Kubernetes identity and KUBECONFIG files.   List  "namespaced" objects. kubectl api-resources -o wide kubectl api-resources --namespaced=false kubectl api-resources --namespaced=true   List objects in the "cluster-local" scope: kubectl api-resources --verbs=list -o name   | xargs -n 1 kubectl get --show-kind --ignore-not-found -l "sas.com/admin=cluster-local" -n lab # in all resources that support the "list" request verbs     Find more articles from SAS Global Enablement and Learning here. ... View more

We already know that the IaC (Infrastructure as Code) tool for Open-Source Kubernetes can be applied to a bunch of on-premise Linux machines and automatically create a Kubernetes cluster using the Upstream Opensource Kubernetes distribution. The Kubernetes cluster will have a control plane plus Kubernetes worker nodes.   But the IaC tool can also be used to provision other "on-prem" infrastructure components such as the Jump Server, the NFS Server or an external PostgreSQL Server.   In the first part of this  series, we’ll focus on why and how to provision and an external PostgreSQL Server for a SAS®  Viya deployment in Opensource Kubernetes using the IaC tool. As recommended by SAS® we will secure the connection between SAS®  Viya and the external database using TLS encryption.   Then in the second part of the blog series, we'll discuss how to connect SAS® Viya to the external PostgreSQL server when we deploy with the viya4-deployment GitHub tool (aka DaC), then open up new perspectives and discuss some additional considerations (HA, Resources consumption impact, PGBouncer, etc…).   Why using an External PostgreSQL ? In the SAS®  Viya platform, an instance of PostgreSQL is required for the SAS® Infrastructure Data Server (Platform PostgreSQL). Several SAS®  Viya platform offerings also require a second Postgres instance referred to as "SAS®  Common Data Store" (or CDS). The CDS cluster and the SAS Infrastructure Data Server cluster must be of the same type: internal or external. In other words, the two clusters must match.  But first let’s clarify what we mean by "Internal" and "External" PostgreSQL (which can be confusing terms).   As noted in the official documentation "Internal" is when you let the PostgreSQL instance be automatically deployed by SAS®  (A containerized Postgres implementation is deployed inside the same Cluster’s namespace as SAS®  Viya and managed by the Crunchy Data Operator ). “External” is when you provide your own PostgreSQL instance.   The decision between using an external versus internal PostgreSQL instance is one of the key Architecture decision to be taken during the Architecture design phase of the project. While using internal Postgres, which is currently the OOTB (or default) choice, seems to be the choice of simplicity, several reasons could lead to chose the external PostgreSQL option: Lack of full support for newer versions of Kubernetes (while SAS®  Viya stable 2023.08 is supported in Kubernetes 1.27, SAS®  recommends using an external instance of PostgreSQL if you want to deploy with Kubernetes 1.27) Automated tuning of some postgres parameters by Cloud providers (Azure, AWS) Advanced PostgreSQL monitoring and log analysis with Cloud Providers services or PostgreSQL community tools  Enhanced backup, disaster recovery features (like automated snapshot backups in Azure, Barman, pgBackRest and other tools designed by the PostgreSQL community to provide data protection etc... One key benefit of running the PostgreSQL instance outside of the Kubernetes cluster  is the removal of the version dependencies between PostgreSQL and Kubernetes, each of these components having their own release schedules.   So let’s see how to use an external PostgreSQL instance for a SAS® Viya deployment in open-source Kubernetes.  😊   Provision the external PostgreSQL with the IaC tool   PostgreSQL installation variables   Each IaC tool has an option to automate the provisioning of an external PostgreSQL System.   But while the IaC tool for the cloud providers (IaC for Azure, IaC for GCP, IaC for AWS) provision it as a Managed Cloud Service (Azure PostgreSQL Flexible Server, Cloud SQL for PostgreSQL, Amazon RDS for PostgreSQL), the IaC tool for Upstream Opensource Kubernetes simply installs the Ubuntu packages of the Open-Source PostgreSQL Server on a Stand-alone machine or a VM that you have to reference in your inventory file.   The variables used by the IaC Tool for Opensource Kubernetes to install and configure an external PostgreSQL Server are documented in the CONFIG-VARS.md file.   Here is a screenshot of the variables name and descriptions associated to the external PostgreSQL (for the "bareOS" mode in the viya4-iac-k8s version 3.0.0 GitHub project).     Notes :   The postgres_ip is not required, you can use the postgres_server_name value instead and use a FQDN. It is highly recommended to secure your communications with the PostgreSQL Server through TLS encryption (postgres_server_ssl="on"). You can set your own SSL certificate and SSL key file with the postgres_server_ssl_cert_file and postgres_server_ssl_key_file variables, but if you don’t specify them, the system default certificate is used by the IaC tool.   Example   Here is an example of the required section, that you need in your inventory file, if you want the IaC for K8s tool to provision the PostgreSQL server for you :   [test_k8s_default_pgsql] rext03-0038.race.sas.com [test_k8s_default_pgsql:vars] postgres_server_name="default" postgres_server_version="14" postgres_server_ssl="on" postgres_administrator_login="postgres" postgres_administrator_password="admin123" #postgres_server_ssl_cert_file="" #postgres_server_ssl_key_file="" postgres_system_setting_max_prepared_transactions="1024" postgres_system_setting_max_connections="1024" [postgres:children] test_k8s_default_pgsql   Note that for some Viya offerings/solutions, you also need an extra PostgreSQL server instance for the SAS® Common Data Store (CDS). You can also provision it (as an external instance) with the IaC tool. You would simply need to add a new host group, like [cds_k8s_pgsql] with the same associated variables in the [cds_k8s_pgsql:vars] section and finally you would reference it as a second item in the [postgres:children] host group.   Challenges and requirements   If you deploy PostgreSQL on an Ubuntu machine/VM, there are some important system requirements for a successful deployment of the opensource PostgreSQL.   I’ve learned some useful lessons, sometimes the  hard way 😉.   When I tested the PostgreSQL server provisioning with the IaC tool on Ubuntu 22.04,  I noted two issues :   The first one was a failure of the apt install command because of missing packages and dependencies issues. We had to install two additional apt packages (and one of them directly from the .deb file). Then after fixing this first problem, while the installation of the PostgreSQL package by the playbook seemed successful, the next Ansible task of the IaC playbook ("update postgres admin user's password") was always failing. ☹️   To understand the second issue, we need to look at the IaC code of the ansible tasks installing PostgreSQL.   In one of the very first task of the playbook, the apt install command initially installs PostgreSQL with SSL enabled, because it is the default even if you set the variable postgres_server_ssl to OFF in your inventory file.   Then the next task tries to update the postgres administrative user's password, but the assumption is that the PostgreSQL service is already running (with SSL) and is ready for connections at this point...   It was not our case. The PostgreSQL service was always failing with the same error message :   2023-05-16 16:25:44.885 UTC [26549] FATAL: could not load server certificate file "/etc/ssl/certs/ssl-cert-snakeoil.pem": SSL error code 214748366   The log message indicates that the service fails because the file "/etc/ssl/certs/ssl-cert-snakeoil.pem" could not be loaded.   The ssl-cert-snakeoil.pem certificate is the default system certificate file on Ubuntu and it appears to be required in order to have a successful startup of the PostgreSQL service (when installing it as a debian package with the apt install command). But unfortunately, it might not have been installed on your Ubuntu system!   The default system certificate can be installed as part of the ssl-certpackage, the default “snake-oil” certificate also needs to be generated and made readable for the account used to start the PostgreSQL service (“postgres” in my case).   Finally, we were able to get the PostgreSQL server successfully provisioned, when running the following commands (before running the IaC playbook):   # Update the sources and install some required debian packages on the sasnode01 sudo apt update sudo apt install sysstat -y wget http://mirrors.kernel.org/ubuntu/pool/main/l/llvm-toolchain-14/libllvm14_14.0.0-1ubuntu1_amd64.deb sudo apt install ./libllvm14_14.0.0-1ubuntu1_amd64.deb# Install and Generate the default snakeoil certificate sudo apt install ssl-cert -y sudo make-ssl-cert generate-default-snakeoil sudo ls -l /etc/ssl/certs/ssl-cert-snakeoil.pem sudo chgrp ssl-cert /etc/ssl/certs/ssl-cert-snakeoil.pem sudo chmod 640 /etc/ssl/certs/ssl-cert-snakeoil.pem# check sudo ls -l /etc/ssl/certs/ssl-cert-snakeoil.pem   Of course, your mileage may vary and you may not hit this particular issue since every Ubuntu machine could have different setup... But be aware that you may have to deal with this kind of issues and other system configuration. Being able to look into the IaC code to see exactly what is done is usually very useful to troubleshoot and resolve your issue.   Check the PostgreSQL server system settings   After the deployment, you can see a PostgreSQL process running on the VM specified in the inventory file:     There is also a PostgreSQL service (run "sudo systemctl status postgresql" to check).   If you are curious, you might open the configuration file that appears in the process description above ("/etc/postgresql/14/main/postgresql.conf").   If you do so, you’ll probably notice something wrong !       Wait a minute…the parameters do not seem to reflect what we've asked for in our inventory file ! remember we had sepcified:   postgres_system_setting_max_prepared_transactions="1024" postgres_system_setting_max_connections="1024"   Well…actually there is no problem in the IaC tool but there is a trick 😊:   The max_connections and max_prepared_transactions are set by the IaC code using the ALTER SYSTEM SQL command (here is a link to the ansible code).   According to the PostgreSQL doc  : "ALTER SYSTEM is used for changing server configuration parameters across the entire database cluster. It can be more convenient than the traditional method of manually editing the postgresql.conf file. ALTER SYSTEM writes the given parameter setting to the postgresql.auto.conf file, which is read in addition to postgresql.conf."   Ok so that explains why we still had the initial values in the postgresql.conf file 😊. But, really the best way to know the truth is to ask directly to the PostgreSQL Server. Here is how you can do it: from the machine/VM where postgres is installed, run the psql command line tool to connect to your PostgreSQL server:   psql -h localhost -U postgres   Then you have to provide the password (the one that was specified in the inventory file).   You should then see something like that :     Now, you can run a command to look into the PostgreSQL settings and confirm the values for the "max_connections" and "max_prepared_connections" parameters.   SELECT Name,Setting from pg_settings WHERE name='max_prepared_transactions' OR name='max_connections';   Now, you can see that the number of maximum connections and maximum prepared transactions corresponds to what we wanted.     While we are there, we can have a look at the SSL parameters.   SELECT Name,Setting from pg_settings WHERE category='Connections and Authentication / SSL';   You should see something like that :     We’ll see, in the second part of the blog, that this information is useful when we need to configure SAS®  Viya to use this external PostgreSQL Server instance for the SAS®  Infrastructure  Data Server. Conclusion   I hope the information and this configuration first time experience feedback (shared in this first part of the series) was helpful. Note that the SAS Global Enablement and Learning team is developing specific SAS® Viya 4 deployment hands-on content within workshop style content. Contact your local SAS Education team for more details if this is of interest to you.   In this blog we only looked at the provisioning of the PostgreSQL server. But stay tuned for the second part of the series, where we go a little bit further: discuss what is needed to connect securely (with TLS) our SAS® Viya to the PostgreSQL server and discuss additional architecture considerations for the external PostgreSQL around HA, tuning and resource utilization. Find more articles from SAS Global Enablement and Learning here. ... View more

In the first part of this post, after the required reminders on how the pods resources are defined in Kubernetes and what are the pods QoS classes, we reviewed the available options for the CAS resources settings : auto-resourcing, customized values or initial values.   CAS auto-resourcing is the recommended choice and is enabled if you apply the default kustomization.yaml file provided in the official documentation.   With CAS auto-resourcing, we rely on the CAS Deployment Operator to identify the CAS nodes (based on their label), collect "allocatable" resources on the host and determine the appropriate CAS pods resource request and limit values.   Before the stable 2023.02 version, with CAS auto-resourcing enabled, the resource requests and limits were set to the same value (for both CPU and memory) which was approximatively 85% of the CPU and memory capacity of the node. As a consequence the CAS pods were running with the "Guaranteed QoS"  But it is different now...   CAS Auto-resourcing changes   What has changed ?     While the stable 2023.02 What’s new note  “Auto-Resourcing Enhancements” is the first place to find some information about this change, here are some additional explanations on what has changed :   When auto-resourcing is not being used; Nothing has changed With auto-resourcing: We will now set smaller request for cpu and memory, and a larger limit for cpu and memory for the side cars. For the CAS container, we set the request to roughly 75% of the CPU rounded down to the next whole number and with the cpu limit to be 100%. Doing this allows the side cars to boost up in performance when needed (For example on Sunday when the backup agent kicks off) while still allowing the CAS pod to use all the resources when the side cars are not utilizing them. With this change, the QOS of CAS with auto-resourcing is changed from Guaranteed -> Burstable. However, this is ok, in the case of auto-resourcing since CAS will be the only pod running on the node; and so would have been the one OOM killed anyways if an OOM event occurs.”   So when the CAS auto-resourcing is NOT in set in the kustomization.yaml , then things remain the same : Installation engineers can customize the CAS pods resources requests and limits or just leave the initial manifest values.   But if the CAS auto-resourcing is enabled then the behavior is slightly different :   First, the CPU request value is expressed in number of cores and set to roughly 75% of the available CPU (rounded down to the next whole number) on the node and the limit value is set to 100%. Then the memory request value is also set to roughly 75% of the available physical memory (rounded down to the next whole number) on the node and the limit is set to 100%. The last change, which is an automatic consequence of the changes above, is that the CAS pods move from the "Guaranteed" to the "Burstable" QoS class (since they now have different values for requests and limits).     Let’s see an example where the CAS nodes are machines with 8 CPU and 64GB of RAM.     Why did we change it ?   The goal of the resource specification changes in the containers is mostly to accommodate the "cas-backup" agent activities. The change enables supplementary container resources to increase when auto-resourcing is configured, and when a burst of activity is required.   For example, the SAS Viya platform is backed up by default every Sunday morning, when the CAS server typically is not running. With this change, the backup agent and other component containers can now temporarily use a larger share of CPU resources (up to one full core by default) that are available on the node. Backups complete more rapidly as a result.   This tradeoff means the CAS pods do not benefit any longer from the Kubernetes "Guaranteed QoS" anymore (different values for resource and limits).   However if the CAS nodes are tainted (as strongly recommended when using CAS auto-resourcing) it would not be an issue since the CAS pod would be  the only pod running on the node. Remember that the "Guaranteed QoS" provide higher protection (against evictions) of the pod when it is in concurrence with other pods running on the same node.   Finally, another change to note is that when not using the CAS auto-resourcing configuration, we now explicitly recommend in the official documentation to set the resources specification with the "Guaranteed QoS". It prevents the CAS pods to be evicted when running in concurrence with other pods on the same node.   Extra considerations   SAS CAS Control pod tuning The tuning documentation mentions specific resource tuning that could be implemented for the sas-cas-control pod to allow the Viya platform to support an increased number of CAS sessions and avoid running into OOM (Out of Memory issues). Here we are NOT talking about the CAS pods that corresponds to CAS SMP or MPP "nodes" and whose resources are managed by the CAS Deployment operator.    The sas-cas-control pod is a stateless service whose purpose is to assist CAS. Here are its default resources requests and limits :     The sas-cas-control pod has two main roles:    1) A bridge for all applications (Visual Analytics, Model Studio, etc…) to access to the CAS servers in the Viya deployment 2)  For the Consul pod to manage the CAS using the SAS Environment application   What has been observed during various benchmarks is that the sas-control-pod could become a bottleneck under heavy load of the CAS server, when a lot of CAS sessions are started.   We can see from the screenshot that the default CPU and memory limits of the sas-cas-control pod are respectively set to 500 millicores (half a core) and 2500Mi (around 2.5 GB).   So, if a lot of CAS sessions are expected, it would be a good idea to monitor these limits and in case the actual usage come close to the limit, increase them accordingly for higher user concurrencies. Otherwise CAS sessions could be closed unexpectedly.   Monitor your Pods resources with k9s       If (just like me 😊), you like to use k9s to interact with your kubernetes clusters, there is something very useful to monitor the resource utilization.   In the "pods view", several columns provide information on the real time resource consumption and how close the CPU and memory utilization approach their defined limits.   Here are the fields displayed by default :     In the screenshots below, thanks to these fields, you can easily see how the CAS resources utilization evolves, in different situations.   Idle :     Opening report in VA : The CAS Controller is using 1% of the CPU limit, 2% of the memory limit.     Now, when we start to work on Analytics pipelines in SAS Model Studio and let SAS explore a large dataset and automatically generate the pipeline from it, we can start to see a significant increase of the CAS Controller using CPU up to more than 70% of the resource limit that was set for the pod !     When you are looking at all the pods in the same time, you can also sort by the columns, for example, type   <Ctrl+x> to sort by %CPU/L or  <ctrl-q> to sort by %MEM/L.   So k9s is very handy to see, in real time, how the real consumption is approaching the limits.   However, keep in mind that in production environments the good practice is to implement alerts (for example with Prometheus, as explained by Ajmal Farzam in this article) in order to detect when pods resource consumption is getting close to the limits or when nodes resources are under pressure.    Sometimes it is not that simple.   A few weeks ago, this question was asked to our team :   The problem here, is that a CAS worker pod could not be scheduled on the intended CAS node because there are already pods running there and "reserving" a portion of the node resources. But how can it happen if we have tainted our CAS nodes to only allow pods with the “CAS” toleration to go there ???    So, in Kubernetes there is a special type of workload resource called “Daemonsets”.   "A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them." It is very handy for things like a cluster storage daemon that must run on every node or a monitoring agent that should collect metrics on each node, etc…   In order to allow these pods to really run everywhere (including on tainted nodes), the DaemonSet controller sets the "node.kubernetes.io/unschedulable:NoSchedule" toleration automatically. Kubernetes can then run DaemonSet Pods on nodes that are marked as unschedulable. It answers our question above and explain why these pods can even run on our "CAS" tainted node...   So if the "DaemonSets managed" pods were already scheduled on the Node and booking significant CPU/ram resources, then when Kubernetes tries to schedule our CAS pods with very high resource requests (because of the auto-resourcing configuration), it is likely that they could not be satisfied and the CAS pod remains in "PENDING" state.   In such case, you might have to disable the CAS auto-resource and instead, customize the CAS pods resources CPU/memory request manually to the values that would work in your situation.   However, note that with the recent changes in stable 2023.03, where the CPU and memory request are now set around to 75% of the node’s capacity (instead of 85% previously) this problem should be mitigated to some degree.   Conclusion   CAS (Cloud Analytic Services) is a key component of the Viya platform.   It’s the SAS Viya platform’s primary Analytics engine used by various applications such as "SAS Visual Analytics", or "SAS Model Studio" allowing many users to run all kind of advanced analytics pipelines using Machine learning, Gradient Boosting or Neuronal networks model, or to simply navigate or create new dashboards or reports.   The design of CAS puts performance first. This requires reserving as much CPU and memory for CAS as possible so it can process the data quickly, while also remaining a highly available and resilient service.   That’s why it is so important to understand how the CAS system can be configured, monitored (and scaled up if needed) so it can make the most of available resources on the nodes while remaining "protected" against a potential eviction or Out of memory incidents.   Hopefully this small series of posts would have provided some guidance to reach this goal !  As usually any comments, questions are very welcome 🙂   References :  Rob Collum's post : Where SAS Viya Relies on Kubernetes for Workload Placement Quality of service in Kubernetes : https://medium.com/google-cloud/quality-of-service-class-qos-in-kubernetes-bb76a89eb2c6   ... View more

While the majority of the SAS Administrator’s interactions with the SAS Viya infrastructure happens at the Kubernetes level (with tools like kubectl or Lens) or with SAS proprietary tools (such as the SAS Environment Manager or the sas-viya CLI), there could be some situations where an access to the underlying node host is required : troubleshooting, maintenance, system logs collection, etc…   As an example, SAS tech support and field consultants already encountered a few scenarios where getting on the k8s nodes and querying journalctl/kubelet/kernel message logs were necessary (for example to determine if CAS pods are being killed by the OOM killer).   "How do I login to my Viya nodes ?" has become a frequently asked question, the purpose of this blog is to provide some guidance about the best way to do it.   Disclaimer   In the follow up of this blog we will discuss various methods to log into the Viya nodes. But first of all, let’s make things clear :   SAS does not officially support the solutions presented below  in any form. Logging onto your cluster nodes is strongly discouraged. Development and debugging of ones pod should be done via logging and other means, i.e. APIs, etc.   Different ways to login…   The most obvious way that comes to mind is to connect through SSH to the cluster underlying Linux nodes.   If your Kubernetes cluster is running in the Cloud, another potential option could be to leverage the cloud CLI, for example the Azure CLI has a the az vm run-command invoke  commands to submit individual commands or run a script.   However these techniques are generally not very efficient: specific network rules or policies could prevent this type of SSH access from the outside, and the Cloud CLI option is cumbersome, not always possible and is different depending on the cloud provider.   Cloud providers (Azure, AWS, GCP, …) are giving you a "Kubernetes Managed Service" (AKS, EKS, GKE) and usually don’t like you to directly log into the underlying VMs by-passing the natural way to interact with the service. That’s why there are often barriers preventing you to directly get access to the underlying Cloud VM.   In Azure, if you are using the IaC tool (viya4-iac-azure) to provision your AKS infrastructure and have opted for the creation of a jump host VM, then you can use the private ssh key (required for the IaC build) to connect via SSH to the Jump Host VM in the Azure and from there, access the AKS worker nodes (the corresponding public key is distributed and “injected” in the AKS nodes as part of the IaC execution).   But it remains a pretty cumbersome, with a “two hops” process and this solution is not necessarily working for other Kubernetes platforms (not working in AWS EK for example).     But better to use the “container based” way…   However, there is a method that should work for any type of Kubernetes platform and does not require any specific host account password or SSH private key.   This generic solution is what we call the "container based" way because the technique used is to start a new pod with a container and exec into it to get access to the underlying host.   Requirements   We will show and explain two methods : with an open source utility called node-shell or directly using the kubectl command, but whatever the method you choose, the requirements are the same :   The KUBECONFIG file being used to access the cluster must have admin rights to the cluster. A cluster-wide admin level kube config file is required. A namespaced kube config will not work with the information provided. Only this cluster-wide admin level provides the 'root' level access on the nodes. You must also have access from the machine you're running kubectl from a network perspective either VPN, direct, or other means.   Method 1 : node-shell plugin   The first method one relies on a small utility called node-shell that lets you start a root shell in the node's host OS running. If you have been using the Lens application (great UI administration interface for Kubernetes) you are actually already using it without knowing it 😊. Indeed Lens has a feature in the Nodes view to let you open an SSH connection to the Kubernetes nodes. Behind the scene the node-shell utility is used.     The node-shell utility can be pulled and installed either in stand-alone mode, directly with 3 commands or it can be installed through krew .   Krew is a plugin manager for kubectl, it allows you to install various plugin to increase even more the capabilities of the kubectl command.   After having installed krew by following the instructions from there it is very easy to install the node-shell plugin (as well as other nice plugins) :   kubectl krew install node-shell   From here, we simply list our nodes: kubectl get nodes   For example, in azure we would see something like:   NAME STATUS ROLES AGE VERSION aks-cas-17945816-vmss000000 Ready agent 4m54s v1.23.12 aks-compute-27817762-vmss000000 Ready agent 4m30s v1.23.12 aks-stateful-16007966-vmss000000 Ready agent 4m55s v1.23.12 aks-stateless-37479317-vmss000000 Ready agent 4m58s v1.23.12 aks-system-48059844-vmss000000 Ready agent 11m v1.23.12   Then from that output, we simply target a node. For this example we'll say the node we want to connect to is   aks-cas-17945816-vmss000000 :   kubectl node-shell aks-cas-17945816-vmss000000   At this point you should see a prompt from that node, just like if you had effectively "SSHed" into the node.   spawning "nsenter-2njj8b" on "aks-cas-17945816-vmss000000" If you don't see a command prompt, try pressing enter. root@aks-cas-17945816-vmss000000:/# id uid=0(root) gid=0(root) groups=0(root),1(daemon),2(bin),3(sys),4(adm),6(disk),10(uucp),11,20(dialout),26(tape),27(sudo) root@aks-cas-17945816-vmss000000:/# ls NOTICE.txt bin boot dev etc home initrd.img initrd.img.old lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var vmlinuz vmlinuz.old root@aks-cas-17945816-vmss000000:/# Method 2 : kubectl debug   If you don’t want (or are not allowed) to install anything on your client/jump host machine in addition to kubectl, you can use a kubectl native command called kubectl debug .   This feature is provided out of the box and appears in the Kubernetes official documentation. Here is the syntax :   kubectl debug node/<node name> -it --image=<containerized OS>   All you need to provide is the name of the node that you want to connect to and the image of the container used to debug (the image must at least have the bash command).   For example in Azure , it would be something like :   [cloud-user@pdcesx03094 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION aks-cas-17945816-vmss000000 Ready agent 3m37s v1.23.12 aks-compute-27817762-vmss000000 Ready agent 3m13s v1.23.12 aks-stateful-16007966-vmss000000 Ready agent 3m38s v1.23.12 aks-stateless-37479317-vmss000000 Ready agent 3m41s v1.23.12 aks-system-48059844-vmss000000 Ready agent 10m v1.23.12 [cloud-user@pdcesx03094 ~]$ kubectl debug node/aks-cas-17945816-vmss000000 -it --image=busybox Creating debugging pod node-debugger-aks-cas-17945816-vmss000000-m9xm8 with container debugger on node aks-cas-17945816-vmss000000. If you don't see a command prompt, try pressing enter. /#   Note that in this example, we’ve been using busybox which is a minimal linux system which does not necessarily include the system debugging tools that you might need to troubleshoot an infrastructure level issue.   Instead, you could use a more complete image that would include additional system debugging tools, such as ubuntu for example, but it would pull a larger image and run a heavier container in the cluster.   This way to get SSH access to the host is the simplest and the most generic, I was able to test it with success for every supported Kubernetes platform.   Example in GCP:     Example in AWS:     Example in opensource K8s:     Example in RHEL OpenShift:     Finally, please note that there are a few important things to know about the kubectl debug command though, here is an extract from the Kubernetes documentation :     Conclusion   As we’ve seen, the two techniques ( node-shell and kubectl debug ) work, no matter what the cloud provider is, how the network has been secured or if host accounts public keys have been "cloud-init" loaded or not.   However, while required in some situation, such type of direct access should remain an exception and should be done very carefully only by the SAS or Kubernetes administrator (not every developer !) for ad hoc debugging or troubleshooting.   As rightfully noted by the SAS R&D:   “This kind SSH access could be abused to do additional node setup, but it is a bad practice since nodes are ephemeral and can come and go with autoscaling etc. Node modifications should be performed via DaemonSets or similar.   While abuse cannot be ruled out, there are still legitimate use cases for access nodes directly, to explore node configurations, to debug issues (e.g., to investigate why the Daemonset does not work as intended) etc.”   That's all, thanks for reading! ... View more