When the SAS Deployment Operator was initially released, one of its key capabilities was to read the SAS Viya configuration from a remote source (such as a source code repository or a Web server).   Today, the latest deployment method, known as the sas-orchestration command also offers this option.   This article discusses the benefits to read the SAS Viya Configuration from a remote git repository (instead of using local files) and explains how to do it with the sas-orchestration command. It also shows an implementation example.   Why?   A SAS Viya deployment configuration is comprised of two main items:   a kustomization.yaml "main" file, and the site-config folder files which are typically stored under the same file system structure on your jump host.   Managing these configuration items can become a bit challenging overtime and make it difficult, to know what has changed and when…   For traceability purposes you could, for example, create a new copy of your SAS Viya configuration folder for each update (each month if you are on a stable cadence, and each time you modify the configuration).   After a little while that historical group of directories might look something like:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   As you can see it is not very efficient; a lot of files are duplicated, and it could quickly become pretty messy…(and I’m not even showing the structure under site-config in the screenshot).   In addition, keeping the SAS Viya configuration on a jump machine’s file system means that you need to keep this machine around, make sure it is secured and backed up, etc… which is not very convenient or aligned with the current DevOps standards (where everything should be ephemeral and automatically built by a pipeline).   On the other hand, having your SAS Viya configuration files stored in a modern source code repository system (such as GitLab or GitHub) brings all the benefits of a modern code versioning system. You could organize your environments in different branches and keep track of all the changes, tag your deployed version and easily come back to them, etc…   It would also be much easier to integrate the SAS Viya configuration in a modern DevOps system.   Deploy SAS Viya from a remote location   The go-getter project provides a library, used behind the scene by the DaaS* tools, to pull the required SAS Viya configuration from a remote location.   As explained in the go-getter project’s README, “go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input”. Out of the box it supports the following protocols : local files, Git, Mercurial, HTTP, Amazon S3, Google GCS.   So, in theory, as long as you can setup the authentication to the remote repository and figure out the right “go-getter” URL (in the Deployment operator CRD or in the sas-orchestration command line), you could place and manage your SAS Viya configuration in any storage system (providing access through one of these go-getter protocols).   However, the current reality is a bit different 😊   While go-getter is used by the DaaS tools, it does not mean every possible system and authentication combination will automatically work when used from the sas-orchestration tool or Deployment operator.   For example, local files and "Git over SSH" access is currently not supported by the DaaS tools. So far, the only "go-getter" URLs tested and supported are the ones using the "Git over http(s)" protocol, like :   For a public repository : git::http://github.com/jsmith/public.git//folder For a private repository: git::http://jsmith:<git_token>@github.com/jsmith/myviya.git//conf   Also, note that the "Git over http/s" go-getter URL only works for the SAS Viya configuration files ( kustomization.yaml and site-config folder files), but not for the order’s certificate information or the license file.   The DaaS tools can read the order’s certificate information or the license file either from a local file system or from an HTTP server.   (*) DaaS = Deployment as a Service: DaaS tools are the Deployment operator and sas-orchestration command. Using GitLab   In terms of infrastructure, we can install a containerized instance of the Open Source GitLab* platform inside our on-premise (Upstream Open source Kubernetes) or Cloud Managed Kubernetes Clusters (Google GKE, Amazon EKS or Microsoft AKS).   In such case, sometimes for demo or training puproses, it is possible to take some shortcuts, using the default "root" account in GitLab and making the project public, so we don’t have to perform any real authentication in the go-getter Git URL.   But a much better practice is to store and manage the SAS Viya configuration in a private GitLab repository.   So, in the next paragraph, we’ll show an example with the “SAS orchestration deploy command” using a git personal token to authenticate and to read the order’s asset and SAS Viya configuration from a private GitLab repository.   (*) GitLab is a very popular web-based Git repository that provides free open and private repositories, issue-following capabilities, and wikis. An example with GitLab and personal access tokens   Create your GitLab token   GitLab personal access tokens allow you to authenticate with Git using HTTP Basic Authentication. This is what is expected to be used in the sas-orchestration tool.   Users can easily create their own personal access token from the GitLab web interface.   All you have to do, once you have signed in, is to right click on your GitLab avatar image, select "Edit Profile", then click on "Access Token".     Then you simply give a name to the token, consider changing the expiration date, select a scope ( read_repository and write_repository scopes are sufficient) and click on the blue button "create personal access token".   Once it is done, you should see a message that confirms that "Your new personal access token has been created."     Now, it is important that you take note of the token content (you can reveal it or just copy it in your buffer), because we will need it later when we run the deployment command.   Store your SAS Viya configuration in GitLab   After that, you can create your "Viya" GitLab repository and store your SAS Viya configuration files there. It can be done either with the GitLab web UI or through Git commands.   If you use Git commands, you’ll first have to setup and configure the Git client on your machine, here is an example (on Linux):   git config --global credential.helper store git config --global http.sslVerify false git config --global user.name "GEL Student" git config --global user.username jsmith git config --global user.password "lnxsas123" git config --global user.email "GELStudent@mycompany.com" These commands update the Git configuration file located in the user’s home directory ( ~/.gitconfig ). You can easily check your Git configuration just by running " git config --list ".   In our scenario, we assume that we don't have the SAS Viya configuration in the remote repository yet.   So, we first need to create a new set of  SAS Viya configuration files ( kustomization.yaml and " site-config " directory) in a local folder.   Once they are in the local folder, we can initialize the repository, reference the remote location, perform our first commit (and finally push to our remote GitLab repository).   You can find below an example of the commands to do that (on a linux machine).   # move to the local repo folder (that contains our SAS Viya configuration files) cd ~/project/deploy/viyaorch-repo # Initialize the git repository git init # Reference the remote location (it also creates the project in GitLab). git remote add origin http://gitlab.devops.company.com/jsmith/viyaorch-repo.git # Now commit our changes in the local git repository. git add . git commit -m "Initial commit of the viyaorch configuration."   You would see something like that:     At this point the Git repository has been initiated but it remains local.   Now it is time to "push" our changes toward the remote GitLab repository.   git push http://gitlab.devops.company.com/jsmith/viyaorch-repo.git     The first time, you are asked for your git username and password*, but then your credentials (that are in your .gitconfig file) are stored  for the next time (remember the credentials.helper parameter was set to store ).   Back to the GitLab web interface, you should then be able to see your SAS Viya configuration files in your GitLab project’s repository:     Congratulations, your SAS Viya Configuration is now stored in GitLab ! 🙂       (*) If you want to avoid this first time prompt, you can create a ~/.git-credentials file in advance with the credential in the following format :  "http://<git username>:<git password@gitlab.devops.company.com" Deploy SAS Viya   The final step is to set up the go-getter URL in the sas-orchestration command and run it to deploy SAS Viya!   In the user-content parameter for the sas-orchestration command, we specify the GitLab personal access token that we created earlier (instead of the user’s GitLab password).   Here is an example:   docker run --rm \ -v $(pwd):/workingdir/ \ -v ~/.kube/config:/kube/config \ -e "KUBECONFIG=/kube/config" \ --user $(id -u):$(id -g) \ sas-orch \ deploy \ --namespace viyaorch \ --deployment-data /workingdir/viyaorch/SASViyaV4_certs.zip \ --license /workingdir/viyaorch/SASViyaV4_license.jwt \ --user-content "git::http://jsmith:<gitlab personal access token>@gitlab.devops.company.com/jsmith/viyaorch-repo.git//config" \ --cadence-name stable \ --cadence-version 2023.06   Note that in the example we still read the order’s certificate ( SASViyaV4_certs.zip ) and license ( SASViyaV4_license.jwt ) files from the local filesystem in this case (but they could also be read from a remote HTTP server).   Finally, if you would like to experiment by yourself, note that the SAS Global Enablement and Learning team is developing specific SAS® Viya 4 deployment hands-on content within workshop style content. Contact your local SAS Education team for more details if this is of interest to you.   Cloud Git Repositories alternatives   While it makes sense in an "on-premise" environment to run a local GitLab instance, for deployment in the Public Cloud the use of the Cloud Provider code repository solution could be a better and more consistent option in the long run.   Your customer might even have this as a requirement if the Cloud Provider code repository is part of their standards and is already used for other projects.   Azure, AWS and Google Cloud Platform offer “git compatible” repository services, such as  Azure Repos, AWS CodeCommit or Google Cloud Source Repositories.   As explained before, while go-getter can used by the DaaS tools (for the "git over http(s)" protocol), it does not mean every possible Cloud Provided Git Repository system and authentication combination will work automatically when used from the sas-orchestration tool or Deployment operator.   However, in the next article we will show how (with some tweaks and tricks) the Google Cloud Source Repositories service can be used as the SAS Viya configuration source for a SAS Viya deployment in the Google Cloud Platform.     Thanks for reading! ... View more

In the first part of the series, we focused on the reasons to choose an external PostgreSQL instance for the SAS Viya Data Infrastructure Server and how to provision it with the IaC tool for Kubernetes.   In this 2 nd article, we detail how the viya4-deployment GitHub project (aka DaC tool) can now be configured to deploy SAS Viya (with an external PostgreSQL server).   Then we'll open up on additional considerations around the utilization of an external PostgreSQL instance for the Viya platform (HA, connections, tuning , technical benefits).   Finally, in the last section of the article, we compare the Kubernetes cluster CPU and memory resource usage and reservation between two environments (one using the internal Crunchy Data Server and one using the external PostgreSQL instance running on a distinct VM).   How to configure SAS Viya for it (with the DaC tool)   The option to use an external PostgreSQL server as the SAS Infrastructure Data Server has been available for some time.   This excellent article from Edoardo Riva is the perfect starting point to understand how to configure Viya to use it.   Basically, you have to provide your Viya environment with some connection information and credentials, so all the Viya components that need to interact with the PostgreSQL Server database (write or read data from the SAS Infrastructure Data Server) can do so.   Here, we just look at an example in the case of an Opensource Kubernetes platform with the most natural companion to the IaC, the viya4-deployment tool (aka "DaC" for "Deployment As Code").   PostgreSQL Server details   The variable used to define the PostgreSQL details (internal or external) in the DaC tool is named V4_CFG_POSTGRES_SERVERS and is actually a "map" of objects.   The variables documentation page shows the various available options, such as the server FQDN and port, the database name, the size of the internal PostgreSQL and pgBackrest PVCs, etc…   Note that several SAS Viya platform offerings require a second internal Postgres instance referred to as SAS Common Data Store or CDS PostgreSQL (the CDS PostgreSQL can also be provisioned by the IaC tool, as noted in the first part of the series).   Here is an example for an external PostgreSQL server :   # External servers V4_CFG_POSTGRES_SERVERS: default: internal: false admin: pgadmin password: "password" fqdn: mydbserver.local server_port: 5432 ssl_enforcement_enabled: true database: SharedServices other_db: internal: false admin: pgadmin password: "password" fqdn: 10.10.10.10 server_port: 5432 ssl_enforcement_enabled: true database: OtherDB Trusted certificate for the PostgreSQL Server   In order to encrypt the communications between the Viya "clients" (all services needing to persist or request information in the SAS Infrastructure Data Server) and the PostgreSQL service we also need to provide the PostgreSQL certificate file location, so the DaC tool can add it in the SAS Viya trust store.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The PEM-encoded certificate file that we need is on the PostgreSQL server and, as shown in the first part of the article, its location can be found in the PostgreSQL settings.   SELECT Name,Setting from pg_settings WHERE category='Connections and Authentication / SSL';   So, in this case the V4_CFG_TLS_TRUSTED_CA_CERTS variable in the DaC ansible vars file should point to the file located on the external PostgreSQL server in /etc/ssl/certs/ssl-cert-sas-rext03-0038.pem .   External PostgreSQL Considerations   HA PostgreSQL   The SAS Infrastructure Data Server component is a critical component of the Viya platform… if, at some point, it fails or is not working properly, the entire Viya platform stop working as designed.   So, running it as a "stand-alone" service would introduce a major SPOF (Single Point Of Failure) in the Viya platform architecture.   One of the benefits of the internal PostgreSQL server implementation with the Crunchy Data PostgreSQL operator is that we automatically have a clustered installation of PostgreSQL (1 primary and 2 replicas) which makes the SAS Infrastructure Data Server component Highly-Available.   Now the question is How can we have the same High-Availability with external PostgreSQL ?   There are various possibilities.   The external PostgreSQL implementation also matters.   As of today, according to the official  documentation we support the following options for the external  Postgres implementation:   PostgreSQL (Open Source) Microsoft Azure Database for PostgreSQL - Flexible Server Amazon RDS for PostgreSQL GCP Cloud SQL for PostgreSQL If you decide to use one of the supported Cloud managed external PostgreSQL database, you need to check their current documentation. However, they generally propose options for High Availability (HA) configurations with automatic failover capabilities.   For example in Azure, with the PostgreSQL Flexible server, “When high availability is configured, flexible server automatically provisions and manages a standby. Write-ahead-logs (WAL) is streamed to the replica in synchronous mode using PostgreSQL streaming replication.”   An alternative is to use the Open Source version of PostgreSQL.   It can be installed on a standalone VM. And that’s exactly what is done when using the IaC for Kubernetes tool. But it would not provide any High Availability by default. In order to have HA, the PostgreSQL should be deployed in a cluster of at least 3 machines and be complemented with something like PGPool for the request load-balancing and a virtual IP address (this is similar  to the Viya 3.5 architecture). Implementing such kind of setup is not trivial.   Another possibility would be to rely on Kubernetes itself to provide this HA capabilities by deploying  PostgreSQL with an operator. The operator could be deployed in the same Kubernetes cluster as Viya.   Here is a list of PG Operators implementation that could potentially be used:   Crunchy Data Zalando CloudNativePG Percona Stackgres  (credits to @Carus Kyle for providing this list 😊)   While the experience and feedback on these options are still limited, they will likely allow the installation of a Highly-Available PostgreSQL server. It would be up to the customer team to ensure the ongoing operation of this PostgreSQL server. While these implementations are opensource, most of them (such as CloudNativePG or Percona) can come with a "full support" model provided by specialized vendors.   Pgbouncer   One of the challenges when using a Cloud Managed Service for the external PostgreSQL is to ensure that the maximum number of connections to the database is high enough for SAS Viya (we have a lot of microservices that need to open connections with the PostgreSQL database).   As stated in the official requirements : “An external PostgreSQL server should support a maximum number of connections, max_connections on some providers, and max_prepared_transactions of at least 1024.”   Cloud providers sometimes apply limits and thresholds depending on the instance type that underpins the database, including on “max_connections”. For example, it used to be problematic with the Azure Database for PostgreSQL - Single Server as higher (and very costly) pricing tier was required to meet this requirement.   Today though, the default Azure Database for PostgreSQL implementation is "Flexible Server" and it supports up to 5000 connections. It is much better, however even with 5000 connections, it might not be enough for  some  large multi-tenant environments.   That’s where PGBouncer can help. You might have heard about it, it is also briefly mentioned in our SAS Viya tuning guide.   PGBouncer is a connection pooler for PostgreSQL. It allows a larger number of frontend connections to share a smaller pool of backend connections. The aim of PGBouncer is to lower the performance impact of opening new connections to PostgreSQL, by re-using database connections.   Finally, note that Kubernetes Operators for Postgres, such as Zalando usually include the PGBouncer component by default.   Technical Benefits?   There are several benefits in the utilization of an external PostgreSQL implementation for the SAS Data Infrastructure server. Here is a screenshot of a colleague’s comment (in an internal discussion about pros and cons of the "external postgres" decision) that I found quite enlightening :     Finally, from my perspective I was also curious to know if, there was a significant reduction for the Kubernetes Hardware requirement when the SAS infrastructure Data server was located outside of the Kubernetes cluster? So it made me wonder “What is the impact on the cluster resources usage of running PostgreSQL outside of the Kubernetes cluster ?”   Well, actually it is the question that I have tried to answer in the next section 😊   Resource consumption comparison   When running the OOTB Crunchy Data Server in the Viya namespace, we have 3 PostgreSQL pods corresponding to our clustered instances as well as the Postgres Operator pod and a PgBackRest instance pod when backups are running.   So, surely replacing that with an external PostgreSQL outside of the cluster will reduce the overall Viya CPU and memory footprint on the cluster itself !?   Since I was very curious about that, I made some comparisons using k9s and some of the Grafana Dashboards coming with the  SAS Viya 4 Monitoring for Kubernetes.   I deployed the same Viya order in 2 Upstream Opensource Kubernetes environments : one was deployed with the internal Crunchy Data Server and the other was using an extra machine with an external PostgreSQL Stand-alone deployment (outside of the Kubernetes cluster).   Please, note that these are very simple, "one time test", and the values reported there were varying overtime. Also the comparisons are made when the system idle, as we are not generating user activity.   It is clearly not something as thorough and systematic as an official benchmark done by our Performance teams at SAS. However, I found the results quite insightful and I believe it gives a rough idea of the differences in terms of resource reservation and utilization.   Kubernetes objects, overall CPU and memory utilization   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server ("External PostgreSQL")     As we can see with the k9s "pulses" view, running PostgreSQL outside of the cluster reduces the number of Kubernetes objects (Pods, ReplicaSets, StatefulSets, PersistentVolumes,). In our case the overall impact of the CPU and memory used is low, however it is one data point for a single measure in an idle and empty environment.   Overall resources usage   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server (“External PostgreSQL”)     While the overall cluster differences in terms of usage for the CPU and memory are not really significant (around 1%), there is a slightly bigger difference in the resource requests commitments (49.9% vs 53.2% for the CPU). Less CPU and memory resource are "reserved" on the nodes when the external PostgreSQL Server is used.   CPU utilization for the Viya namespace   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server ("External PostgreSQL")     Now if we specifically zoom on the Viya namespace ("dac"), we clearly see the difference in the CPU Requests between the 2 environments (9.23% vs 6.57%).   MEMORY utilization for the Viya namespace   With Crunchy Data Server ("Internal PostgreSQL")     Without Crunchy Data Server ("External PostgreSQL")     Finally, the two Grafana graphs confirms a relatively low difference on the memory usage in an idle/empty environment.   Here are the main takeaways of this first and quick comparison : The amount of CPU and memory resources reserved on the Kubernetes is reduced when PostgreSQL is running outside of the Kubernetes cluster (each of the four Crunchy Data PostgreSQL Instance requests almost half of a CPU and 3GB of memory), ...but it remains a relatively small reduction in the overall Viya Platform resources requirements. In an idle, freshly deployed environment without any user’s activity, there is not significant difference in the CPU and memory utilization between the two environments. Conclusion   And that’s all for this time.   I hope this  series was useful to reiterate some of the established architecture principles and configurations for the SAS infrastructure Data Server component … Thanks for reading ! ... View more

A few weeks ago, I was contacted by my colleague Frederik (from the Global Technology Practice) as his large UK customer was challenging him on our requirements, in terms of permissions in Kubernetes for the deployment and administration of SAS Viya.   It made me realize that when we deploy SAS Viya internally at SAS or in our Deployment or Administration lab environments, we often do it with the Kubernetes administrator full access. With this kind of access, we can create but also delete any cluster-wide resource, such as someone else’s namespace, someone else persistent volume (where critical information might be stored), etc.   While it is very handy and make our life easier each time we need to deploy/redeploy or change anything in SAS Viya, it is quite unlikely the Kubernetes administrator, working for your customer, accepts to give you such kind of access (especially if the Kubernetes cluster is shared with other applications).   Instead, they will probably tell you: "Okay SAS admin, now could you tell me exactly what kind of RBAC* authorizations you need in my Kubernetes cluster to deploy your application?"   If you have experienced this situation or think you may face it sooner or later, you might find this blog useful. Its goal is to propose an approach where the Kubernetes administrator can delegate the administration of the SAS Viya environment to someone with limited privileges in the cluster.   (*) RBAC stands for Role Based Access Control, you can learn more about it there   Review of the existing resources   Viya ARK   The Viya ARK pre-installation playbook checks Creation/Deletion of: CRD, Role, RoleBinding, Deploy, Service, Ingress/Gateway objects.   For example, here is how the "Viya ARK Pre-Install check" report looks like when using a restricted service account (who is a namespace admin and has some additional cluster-wide read-only permissions on namespaces and PVs).   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The report is consistent, since to perform the whole deployment, Cluster admin permissions are required. However, it does not provide answers to the customer questions discussed in the introduction of this post.   Official Documentation   In February 2022, a new section “Cluster Resources and Roles That Require Elevated Permissions” was added in the SAS Operations documentation to provide additional guidance on this topic.   While the doc section remains at a high level (and could be confusing to someone who is new to the Kubernetes RBAC model), it explains that the Kubernetes cluster administrator could create additional roles for delegated administrators who could then deploy and administrate the software.   One of the recommendations is to provide “a delegated, namespace-level administrator with the "get", "list," and "watch" permissions on all resources cluster-wide.”   With this level of permissions, the delegated administrator (let’s call him the SAS Viya administrator) can see all resources (including secrets) in every namespace of the cluster. It is not something that will always be acceptable for customers…and as we’ll see with practical examples, it is not really required.   Challenge and suggested approach   When the SAS consultant cannot have the cluster administration privileges to perform the SAS Viya deployment and administration, the following approach could be followed to address the customer’s requirement.   Use the manual deployment method. Create a "SAS Viya admin" service account that will be used by the SAS consultant. Give permissions to the "SAS Viya admin" service account in the SAS Viya namespace. Split the initial manual deployment commands (depending on their "scope") between the Kubernetes Administrator and the SAS consultant. Only if needed, add specific permissions as needed for additional administration tasks.   What if I want to use an orchestrated deployment method ?   Important : Splitting the initial deployment tasks between the K8s admin and the SAS Consultant (for example with a namespace scoped privileges) is only possible with the manual deployment method.   If the chosen deployment method is the sas-orchestration, then the "cluster level" privileges are required each time the software has to be redeployed…which is something that happens each time you need to update your kustomization.yaml file and force SAS Viya to pick up configuration changes.   If the chosen deployment method is the SAS Viya Platform Deployment Operator, then the re-deployment is always managed through a reconciliation process that is handled by the deployment operator and run under the Deployment Operator's Cluster-role.   Scopes and sas.com/admin labels   The SAS Viya platform is made up of various Kubernetes objects.   Some objects have a “cluster-wide” scope (they are global for the whole cluster and not limited to a specific namespace), others have a "namespace" scope.   In the SAS Viya manifest (site.yml), each object is in a category (or scope) determined by a "sas.com/admin" label.   There are 4 categories (listed in the table below that comes from the SAS official documentation). However, these categories don’t necessarily match what Kubernetes considers as "namespaced" object (Configmaps, Pods, PodTemplates, Services, etc..) and "non-namespaced" objects (Nodes, PersistentVolumes, ClusterRoles, clusterRolebinding, etc…). As an example, apart from one exception, the "cluster-local" category is associated to "namespaced" objects.     The selectors "sas.com/admin=cluster-wide" and "sas.com/admin=cluster-api" are associated to Cluster level objects.   It means that the kubectl apply command, that target these kinds of objects, can only be executed by someone with the default  cluster-admin ClusterRole bound with a ClusterRoleBinding (which gives privileges at the cluster level – i.e. your almighty Kubernetes Administrator).   The selector "sas.com/admin=namespace" is associated with "namespace level" objects. So, someone with a namespace administrator role for the specific namespace, could run the kubectl apply command that targets these objects.   Then we have the "sas.com/admin=cluster-local" level, which is a mixed bag of various kinds of objects that also exist in the other categories. Since there is generally at least one object with this label that is at the cluster-wide level, a namespace administrator role would NOT be able to run the  kubectl apply command that target the "cluster-local" scope with full success…   The issue with the "cluster-local" label   This selector is a bit more problematic since it contains 99% of namespace level object ("configmaps", "secrets", "persistentvolumeclaims", "rolebindings", "podtemplates") but also potentially cluster-wide objects (ClusterRoleBinding for the SAS Workload Orchestrator).   Actually, in our test environment (with the "SAS® Viya®" offering), the only object that prevented us successfully applying the "sas.com/admin=cluster-local" level as a namespace administrator is the "sas-workload-orchestrator" ClusterRoleBinding object.   The issue with this "cluster-local" label is that : if for a given Viya configuration/customization, we need to modify an object that is at this level, then it will be necessary to run the kubectl apply command at the "cluster-local" level…So our namespace level Viya administrator will have to call the Kubernetes admin just to run the kubectl apply at this level (even if the modified object is at the namespace level ! ) ☹️   Removing the "cluster-local" label and assigning the associated resources either to "cluster-wide" or "namespace" categories, would greatly simplify things as it would allow Viya administrators with namespace privileges to perform almost any kind of Viya configuration/customization.   Required role and additional permissions   A recommendation of the official SAS Documentation is to provide "a delegated, namespace-level administrator with the "get", "list," and "watch" permissions on all resources cluster-wide."   Ideally, the namespace administration privileges would be enough (without the need to give permission on all resources cluster-wide). What we can do is to use one of the Kubernetes existing "built-in" roles.   As noted in the official Kubernetes RBAC documentation,  the "cluster-admin" built-in ClusterRole "when used in a RoleBinding, gives full control over every resource in the role binding's namespace, including the namespace itself."   Here is an example on how to give this role to a "viya-admin-lab" service account.   # create the manifest tee /tmp/create-binding-for-viya-admin-lab.yaml > /dev/null << EOF --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: sas-rolebinding-namespace namespace: lab subjects: - kind: ServiceAccount name: viya-admin-lab namespace: lab roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io EOF # apply the manifest kubectl apply -f /tmp/create-binding-for-viya-admin-lab.yaml   We simply create a "RoleBinding" object to attach the built-in "cluster-admin" ClusterRole to our "lab" namespace. So, while it is a "ClusterRole", it can be limited to the scope of a particular namespace.   With the above privilege definition, someone using our "viya-admin-lab" service account could change everything inside our Viya namespace but could not do anything else outside of the namespace.   Note: Here we assume that we have already created the "viya-admin-lab" service account and generated the associate KUBECONFIG file.   Break down the responsibilities   The table below shows the breakdown of tasks between the Kubernetes administrator and our SAS Viya administrator using the service account described above (with his privileges limited to the SAS Viya namespace).   Task Responsibility Notes Deploy SAS Viya Shared The Kubernetes administrator runs the "kubectl apply" commands for "cluster-api", "cluster-wide" and "cluster-local" levels. The SAS Viya administrator runs "kubectl apply" command for the namespace level (and potentially for the "cluster-local" level). Update SAS Viya Shared Stop/Start the SAS Viya platform SAS Viya administrator (namespace admin) This task is handled by Cronjobs defined at the namespace level. Backup and restore the SAS Viya environment SAS Viya administrator (namespace admin) Perform common customizations (change CAS or SAS programming runtime settings, change CAS topology, etc…) SAS Viya administrator (namespace admin) While the SAS Viya administrator cannot change the Custom Resource Definitions (CASDeployment definition), he can modify Custom Resource instances (CASDeployment instances), which is enough to change the configuration of a deployed CAS Server. Allocate additional permissions to the SAS Viya administrators when/if needed for specific configuration. Kubernetes Administrator We haven’t tested all possible configuration/customization. Some may exceptionally require additional privileges. The additional permissions would be added as required.     Note: The described tasks have all been tested in the GEL Deploying SAS Viya workshop Hands-on.     Conclusion   Hopefully this post helped you to understand which kind of authorization is required in Kubernetes for a SAS Consultant or a SAS Administrator to accomplish his tasks related to the SAS Viya deployments.   The content presented here comes from practical experimentation in a Viya LTS 2023.10 environment deployed on an Upstream Opensource Kubernetes cluster (in version 1.27). If you want to see these concepts by yourself in practice, note that the SAS Global Enablement and Learning team is developing specific SAS® Viya 4 deployment hands-on content within workshop style content. Contact your local SAS Education team for more details if this is of interest to you.,.   Bonus: Some useful commands to check permissions and scope   Finally, as a little bonus, you can find below some useful commands to explore the Kubernetes RBAC implementation.   "kubectl-whoami" and "kubectl auth can-i" commands. # the whoami plugin can be installed with krew () kubectl-krew install whoami kubectl whoami # can I create pods ? kubectl auth can-i create pod -n lab # can I create the resources corresponding to the "sas.com/admin=namespace" label in the SAS Viya manifest ? for res in service deployment replicaset replicationcontroller job cronJob statefulset daemonset podtemplate do # test the create privilege for each resource echo "testing creation right on $res" kubectl auth can-i create $res -n lab done   See Rob Collum’s post for more about Kubernetes identity and KUBECONFIG files.   List  "namespaced" objects. kubectl api-resources -o wide kubectl api-resources --namespaced=false kubectl api-resources --namespaced=true   List objects in the "cluster-local" scope: kubectl api-resources --verbs=list -o name   | xargs -n 1 kubectl get --show-kind --ignore-not-found -l "sas.com/admin=cluster-local" -n lab # in all resources that support the "list" request verbs     Find more articles from SAS Global Enablement and Learning here. ... View more

We already know that the IaC (Infrastructure as Code) tool for Open-Source Kubernetes can be applied to a bunch of on-premise Linux machines and automatically create a Kubernetes cluster using the Upstream Opensource Kubernetes distribution. The Kubernetes cluster will have a control plane plus Kubernetes worker nodes.   But the IaC tool can also be used to provision other "on-prem" infrastructure components such as the Jump Server, the NFS Server or an external PostgreSQL Server.   In the first part of this  series, we’ll focus on why and how to provision and an external PostgreSQL Server for a SAS®  Viya deployment in Opensource Kubernetes using the IaC tool. As recommended by SAS® we will secure the connection between SAS®  Viya and the external database using TLS encryption.   Then in the second part of the blog series, we'll discuss how to connect SAS® Viya to the external PostgreSQL server when we deploy with the viya4-deployment GitHub tool (aka DaC), then open up new perspectives and discuss some additional considerations (HA, Resources consumption impact, PGBouncer, etc…).   Why using an External PostgreSQL ? In the SAS®  Viya platform, an instance of PostgreSQL is required for the SAS® Infrastructure Data Server (Platform PostgreSQL). Several SAS®  Viya platform offerings also require a second Postgres instance referred to as "SAS®  Common Data Store" (or CDS). The CDS cluster and the SAS Infrastructure Data Server cluster must be of the same type: internal or external. In other words, the two clusters must match.  But first let’s clarify what we mean by "Internal" and "External" PostgreSQL (which can be confusing terms).   As noted in the official documentation "Internal" is when you let the PostgreSQL instance be automatically deployed by SAS®  (A containerized Postgres implementation is deployed inside the same Cluster’s namespace as SAS®  Viya and managed by the Crunchy Data Operator ). “External” is when you provide your own PostgreSQL instance.   The decision between using an external versus internal PostgreSQL instance is one of the key Architecture decision to be taken during the Architecture design phase of the project. While using internal Postgres, which is currently the OOTB (or default) choice, seems to be the choice of simplicity, several reasons could lead to chose the external PostgreSQL option: Lack of full support for newer versions of Kubernetes (while SAS®  Viya stable 2023.08 is supported in Kubernetes 1.27, SAS®  recommends using an external instance of PostgreSQL if you want to deploy with Kubernetes 1.27) Automated tuning of some postgres parameters by Cloud providers (Azure, AWS) Advanced PostgreSQL monitoring and log analysis with Cloud Providers services or PostgreSQL community tools  Enhanced backup, disaster recovery features (like automated snapshot backups in Azure, Barman, pgBackRest and other tools designed by the PostgreSQL community to provide data protection etc... One key benefit of running the PostgreSQL instance outside of the Kubernetes cluster  is the removal of the version dependencies between PostgreSQL and Kubernetes, each of these components having their own release schedules.   So let’s see how to use an external PostgreSQL instance for a SAS® Viya deployment in open-source Kubernetes.  😊   Provision the external PostgreSQL with the IaC tool   PostgreSQL installation variables   Each IaC tool has an option to automate the provisioning of an external PostgreSQL System.   But while the IaC tool for the cloud providers (IaC for Azure, IaC for GCP, IaC for AWS) provision it as a Managed Cloud Service (Azure PostgreSQL Flexible Server, Cloud SQL for PostgreSQL, Amazon RDS for PostgreSQL), the IaC tool for Upstream Opensource Kubernetes simply installs the Ubuntu packages of the Open-Source PostgreSQL Server on a Stand-alone machine or a VM that you have to reference in your inventory file.   The variables used by the IaC Tool for Opensource Kubernetes to install and configure an external PostgreSQL Server are documented in the CONFIG-VARS.md file.   Here is a screenshot of the variables name and descriptions associated to the external PostgreSQL (for the "bareOS" mode in the viya4-iac-k8s version 3.0.0 GitHub project).     Notes :   The postgres_ip is not required, you can use the postgres_server_name value instead and use a FQDN. It is highly recommended to secure your communications with the PostgreSQL Server through TLS encryption (postgres_server_ssl="on"). You can set your own SSL certificate and SSL key file with the postgres_server_ssl_cert_file and postgres_server_ssl_key_file variables, but if you don’t specify them, the system default certificate is used by the IaC tool.   Example   Here is an example of the required section, that you need in your inventory file, if you want the IaC for K8s tool to provision the PostgreSQL server for you :   [test_k8s_default_pgsql] rext03-0038.race.sas.com [test_k8s_default_pgsql:vars] postgres_server_name="default" postgres_server_version="14" postgres_server_ssl="on" postgres_administrator_login="postgres" postgres_administrator_password="admin123" #postgres_server_ssl_cert_file="" #postgres_server_ssl_key_file="" postgres_system_setting_max_prepared_transactions="1024" postgres_system_setting_max_connections="1024" [postgres:children] test_k8s_default_pgsql   Note that for some Viya offerings/solutions, you also need an extra PostgreSQL server instance for the SAS® Common Data Store (CDS). You can also provision it (as an external instance) with the IaC tool. You would simply need to add a new host group, like [cds_k8s_pgsql] with the same associated variables in the [cds_k8s_pgsql:vars] section and finally you would reference it as a second item in the [postgres:children] host group.   Challenges and requirements   If you deploy PostgreSQL on an Ubuntu machine/VM, there are some important system requirements for a successful deployment of the opensource PostgreSQL.   I’ve learned some useful lessons, sometimes the  hard way 😉.   When I tested the PostgreSQL server provisioning with the IaC tool on Ubuntu 22.04,  I noted two issues :   The first one was a failure of the apt install command because of missing packages and dependencies issues. We had to install two additional apt packages (and one of them directly from the .deb file). Then after fixing this first problem, while the installation of the PostgreSQL package by the playbook seemed successful, the next Ansible task of the IaC playbook ("update postgres admin user's password") was always failing. ☹️   To understand the second issue, we need to look at the IaC code of the ansible tasks installing PostgreSQL.   In one of the very first task of the playbook, the apt install command initially installs PostgreSQL with SSL enabled, because it is the default even if you set the variable postgres_server_ssl to OFF in your inventory file.   Then the next task tries to update the postgres administrative user's password, but the assumption is that the PostgreSQL service is already running (with SSL) and is ready for connections at this point...   It was not our case. The PostgreSQL service was always failing with the same error message :   2023-05-16 16:25:44.885 UTC [26549] FATAL: could not load server certificate file "/etc/ssl/certs/ssl-cert-snakeoil.pem": SSL error code 214748366   The log message indicates that the service fails because the file "/etc/ssl/certs/ssl-cert-snakeoil.pem" could not be loaded.   The ssl-cert-snakeoil.pem certificate is the default system certificate file on Ubuntu and it appears to be required in order to have a successful startup of the PostgreSQL service (when installing it as a debian package with the apt install command). But unfortunately, it might not have been installed on your Ubuntu system!   The default system certificate can be installed as part of the ssl-certpackage, the default “snake-oil” certificate also needs to be generated and made readable for the account used to start the PostgreSQL service (“postgres” in my case).   Finally, we were able to get the PostgreSQL server successfully provisioned, when running the following commands (before running the IaC playbook):   # Update the sources and install some required debian packages on the sasnode01 sudo apt update sudo apt install sysstat -y wget http://mirrors.kernel.org/ubuntu/pool/main/l/llvm-toolchain-14/libllvm14_14.0.0-1ubuntu1_amd64.deb sudo apt install ./libllvm14_14.0.0-1ubuntu1_amd64.deb# Install and Generate the default snakeoil certificate sudo apt install ssl-cert -y sudo make-ssl-cert generate-default-snakeoil sudo ls -l /etc/ssl/certs/ssl-cert-snakeoil.pem sudo chgrp ssl-cert /etc/ssl/certs/ssl-cert-snakeoil.pem sudo chmod 640 /etc/ssl/certs/ssl-cert-snakeoil.pem# check sudo ls -l /etc/ssl/certs/ssl-cert-snakeoil.pem   Of course, your mileage may vary and you may not hit this particular issue since every Ubuntu machine could have different setup... But be aware that you may have to deal with this kind of issues and other system configuration. Being able to look into the IaC code to see exactly what is done is usually very useful to troubleshoot and resolve your issue.   Check the PostgreSQL server system settings   After the deployment, you can see a PostgreSQL process running on the VM specified in the inventory file:     There is also a PostgreSQL service (run "sudo systemctl status postgresql" to check).   If you are curious, you might open the configuration file that appears in the process description above ("/etc/postgresql/14/main/postgresql.conf").   If you do so, you’ll probably notice something wrong !       Wait a minute…the parameters do not seem to reflect what we've asked for in our inventory file ! remember we had sepcified:   postgres_system_setting_max_prepared_transactions="1024" postgres_system_setting_max_connections="1024"   Well…actually there is no problem in the IaC tool but there is a trick 😊:   The max_connections and max_prepared_transactions are set by the IaC code using the ALTER SYSTEM SQL command (here is a link to the ansible code).   According to the PostgreSQL doc  : "ALTER SYSTEM is used for changing server configuration parameters across the entire database cluster. It can be more convenient than the traditional method of manually editing the postgresql.conf file. ALTER SYSTEM writes the given parameter setting to the postgresql.auto.conf file, which is read in addition to postgresql.conf."   Ok so that explains why we still had the initial values in the postgresql.conf file 😊. But, really the best way to know the truth is to ask directly to the PostgreSQL Server. Here is how you can do it: from the machine/VM where postgres is installed, run the psql command line tool to connect to your PostgreSQL server:   psql -h localhost -U postgres   Then you have to provide the password (the one that was specified in the inventory file).   You should then see something like that :     Now, you can run a command to look into the PostgreSQL settings and confirm the values for the "max_connections" and "max_prepared_connections" parameters.   SELECT Name,Setting from pg_settings WHERE name='max_prepared_transactions' OR name='max_connections';   Now, you can see that the number of maximum connections and maximum prepared transactions corresponds to what we wanted.     While we are there, we can have a look at the SSL parameters.   SELECT Name,Setting from pg_settings WHERE category='Connections and Authentication / SSL';   You should see something like that :     We’ll see, in the second part of the blog, that this information is useful when we need to configure SAS®  Viya to use this external PostgreSQL Server instance for the SAS®  Infrastructure  Data Server. Conclusion   I hope the information and this configuration first time experience feedback (shared in this first part of the series) was helpful. Note that the SAS Global Enablement and Learning team is developing specific SAS® Viya 4 deployment hands-on content within workshop style content. Contact your local SAS Education team for more details if this is of interest to you.   In this blog we only looked at the provisioning of the PostgreSQL server. But stay tuned for the second part of the series, where we go a little bit further: discuss what is needed to connect securely (with TLS) our SAS® Viya to the PostgreSQL server and discuss additional architecture considerations for the external PostgreSQL around HA, tuning and resource utilization. Find more articles from SAS Global Enablement and Learning here. ... View more

In the first part of this post, after the required reminders on how the pods resources are defined in Kubernetes and what are the pods QoS classes, we reviewed the available options for the CAS resources settings : auto-resourcing, customized values or initial values.   CAS auto-resourcing is the recommended choice and is enabled if you apply the default kustomization.yaml file provided in the official documentation.   With CAS auto-resourcing, we rely on the CAS Deployment Operator to identify the CAS nodes (based on their label), collect "allocatable" resources on the host and determine the appropriate CAS pods resource request and limit values.   Before the stable 2023.02 version, with CAS auto-resourcing enabled, the resource requests and limits were set to the same value (for both CPU and memory) which was approximatively 85% of the CPU and memory capacity of the node. As a consequence the CAS pods were running with the "Guaranteed QoS"  But it is different now...   CAS Auto-resourcing changes   What has changed ?     While the stable 2023.02 What’s new note  “Auto-Resourcing Enhancements” is the first place to find some information about this change, here are some additional explanations on what has changed :   When auto-resourcing is not being used; Nothing has changed With auto-resourcing: We will now set smaller request for cpu and memory, and a larger limit for cpu and memory for the side cars. For the CAS container, we set the request to roughly 75% of the CPU rounded down to the next whole number and with the cpu limit to be 100%. Doing this allows the side cars to boost up in performance when needed (For example on Sunday when the backup agent kicks off) while still allowing the CAS pod to use all the resources when the side cars are not utilizing them. With this change, the QOS of CAS with auto-resourcing is changed from Guaranteed -> Burstable. However, this is ok, in the case of auto-resourcing since CAS will be the only pod running on the node; and so would have been the one OOM killed anyways if an OOM event occurs.”   So when the CAS auto-resourcing is NOT in set in the kustomization.yaml , then things remain the same : Installation engineers can customize the CAS pods resources requests and limits or just leave the initial manifest values.   But if the CAS auto-resourcing is enabled then the behavior is slightly different :   First, the CPU request value is expressed in number of cores and set to roughly 75% of the available CPU (rounded down to the next whole number) on the node and the limit value is set to 100%. Then the memory request value is also set to roughly 75% of the available physical memory (rounded down to the next whole number) on the node and the limit is set to 100%. The last change, which is an automatic consequence of the changes above, is that the CAS pods move from the "Guaranteed" to the "Burstable" QoS class (since they now have different values for requests and limits).     Let’s see an example where the CAS nodes are machines with 8 CPU and 64GB of RAM.     Why did we change it ?   The goal of the resource specification changes in the containers is mostly to accommodate the "cas-backup" agent activities. The change enables supplementary container resources to increase when auto-resourcing is configured, and when a burst of activity is required.   For example, the SAS Viya platform is backed up by default every Sunday morning, when the CAS server typically is not running. With this change, the backup agent and other component containers can now temporarily use a larger share of CPU resources (up to one full core by default) that are available on the node. Backups complete more rapidly as a result.   This tradeoff means the CAS pods do not benefit any longer from the Kubernetes "Guaranteed QoS" anymore (different values for resource and limits).   However if the CAS nodes are tainted (as strongly recommended when using CAS auto-resourcing) it would not be an issue since the CAS pod would be  the only pod running on the node. Remember that the "Guaranteed QoS" provide higher protection (against evictions) of the pod when it is in concurrence with other pods running on the same node.   Finally, another change to note is that when not using the CAS auto-resourcing configuration, we now explicitly recommend in the official documentation to set the resources specification with the "Guaranteed QoS". It prevents the CAS pods to be evicted when running in concurrence with other pods on the same node.   Extra considerations   SAS CAS Control pod tuning The tuning documentation mentions specific resource tuning that could be implemented for the sas-cas-control pod to allow the Viya platform to support an increased number of CAS sessions and avoid running into OOM (Out of Memory issues). Here we are NOT talking about the CAS pods that corresponds to CAS SMP or MPP "nodes" and whose resources are managed by the CAS Deployment operator.    The sas-cas-control pod is a stateless service whose purpose is to assist CAS. Here are its default resources requests and limits :     The sas-cas-control pod has two main roles:    1) A bridge for all applications (Visual Analytics, Model Studio, etc…) to access to the CAS servers in the Viya deployment 2)  For the Consul pod to manage the CAS using the SAS Environment application   What has been observed during various benchmarks is that the sas-control-pod could become a bottleneck under heavy load of the CAS server, when a lot of CAS sessions are started.   We can see from the screenshot that the default CPU and memory limits of the sas-cas-control pod are respectively set to 500 millicores (half a core) and 2500Mi (around 2.5 GB).   So, if a lot of CAS sessions are expected, it would be a good idea to monitor these limits and in case the actual usage come close to the limit, increase them accordingly for higher user concurrencies. Otherwise CAS sessions could be closed unexpectedly.   Monitor your Pods resources with k9s       If (just like me 😊), you like to use k9s to interact with your kubernetes clusters, there is something very useful to monitor the resource utilization.   In the "pods view", several columns provide information on the real time resource consumption and how close the CPU and memory utilization approach their defined limits.   Here are the fields displayed by default :     In the screenshots below, thanks to these fields, you can easily see how the CAS resources utilization evolves, in different situations.   Idle :     Opening report in VA : The CAS Controller is using 1% of the CPU limit, 2% of the memory limit.     Now, when we start to work on Analytics pipelines in SAS Model Studio and let SAS explore a large dataset and automatically generate the pipeline from it, we can start to see a significant increase of the CAS Controller using CPU up to more than 70% of the resource limit that was set for the pod !     When you are looking at all the pods in the same time, you can also sort by the columns, for example, type   <Ctrl+x> to sort by %CPU/L or  <ctrl-q> to sort by %MEM/L.   So k9s is very handy to see, in real time, how the real consumption is approaching the limits.   However, keep in mind that in production environments the good practice is to implement alerts (for example with Prometheus, as explained by Ajmal Farzam in this article) in order to detect when pods resource consumption is getting close to the limits or when nodes resources are under pressure.    Sometimes it is not that simple.   A few weeks ago, this question was asked to our team :   The problem here, is that a CAS worker pod could not be scheduled on the intended CAS node because there are already pods running there and "reserving" a portion of the node resources. But how can it happen if we have tainted our CAS nodes to only allow pods with the “CAS” toleration to go there ???    So, in Kubernetes there is a special type of workload resource called “Daemonsets”.   "A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them." It is very handy for things like a cluster storage daemon that must run on every node or a monitoring agent that should collect metrics on each node, etc…   In order to allow these pods to really run everywhere (including on tainted nodes), the DaemonSet controller sets the "node.kubernetes.io/unschedulable:NoSchedule" toleration automatically. Kubernetes can then run DaemonSet Pods on nodes that are marked as unschedulable. It answers our question above and explain why these pods can even run on our "CAS" tainted node...   So if the "DaemonSets managed" pods were already scheduled on the Node and booking significant CPU/ram resources, then when Kubernetes tries to schedule our CAS pods with very high resource requests (because of the auto-resourcing configuration), it is likely that they could not be satisfied and the CAS pod remains in "PENDING" state.   In such case, you might have to disable the CAS auto-resource and instead, customize the CAS pods resources CPU/memory request manually to the values that would work in your situation.   However, note that with the recent changes in stable 2023.03, where the CPU and memory request are now set around to 75% of the node’s capacity (instead of 85% previously) this problem should be mitigated to some degree.   Conclusion   CAS (Cloud Analytic Services) is a key component of the Viya platform.   It’s the SAS Viya platform’s primary Analytics engine used by various applications such as "SAS Visual Analytics", or "SAS Model Studio" allowing many users to run all kind of advanced analytics pipelines using Machine learning, Gradient Boosting or Neuronal networks model, or to simply navigate or create new dashboards or reports.   The design of CAS puts performance first. This requires reserving as much CPU and memory for CAS as possible so it can process the data quickly, while also remaining a highly available and resilient service.   That’s why it is so important to understand how the CAS system can be configured, monitored (and scaled up if needed) so it can make the most of available resources on the nodes while remaining "protected" against a potential eviction or Out of memory incidents.   Hopefully this small series of posts would have provided some guidance to reach this goal !  As usually any comments, questions are very welcome 🙂   References :  Rob Collum's post : Where SAS Viya Relies on Kubernetes for Workload Placement Quality of service in Kubernetes : https://medium.com/google-cloud/quality-of-service-class-qos-in-kubernetes-bb76a89eb2c6   ... View more

While the majority of the SAS Administrator’s interactions with the SAS Viya infrastructure happens at the Kubernetes level (with tools like kubectl or Lens) or with SAS proprietary tools (such as the SAS Environment Manager or the sas-viya CLI), there could be some situations where an access to the underlying node host is required : troubleshooting, maintenance, system logs collection, etc…   As an example, SAS tech support and field consultants already encountered a few scenarios where getting on the k8s nodes and querying journalctl/kubelet/kernel message logs were necessary (for example to determine if CAS pods are being killed by the OOM killer).   "How do I login to my Viya nodes ?" has become a frequently asked question, the purpose of this blog is to provide some guidance about the best way to do it.   Disclaimer   In the follow up of this blog we will discuss various methods to log into the Viya nodes. But first of all, let’s make things clear :   SAS does not officially support the solutions presented below  in any form. Logging onto your cluster nodes is strongly discouraged. Development and debugging of ones pod should be done via logging and other means, i.e. APIs, etc.   Different ways to login…   The most obvious way that comes to mind is to connect through SSH to the cluster underlying Linux nodes.   If your Kubernetes cluster is running in the Cloud, another potential option could be to leverage the cloud CLI, for example the Azure CLI has a the az vm run-command invoke  commands to submit individual commands or run a script.   However these techniques are generally not very efficient: specific network rules or policies could prevent this type of SSH access from the outside, and the Cloud CLI option is cumbersome, not always possible and is different depending on the cloud provider.   Cloud providers (Azure, AWS, GCP, …) are giving you a "Kubernetes Managed Service" (AKS, EKS, GKE) and usually don’t like you to directly log into the underlying VMs by-passing the natural way to interact with the service. That’s why there are often barriers preventing you to directly get access to the underlying Cloud VM.   In Azure, if you are using the IaC tool (viya4-iac-azure) to provision your AKS infrastructure and have opted for the creation of a jump host VM, then you can use the private ssh key (required for the IaC build) to connect via SSH to the Jump Host VM in the Azure and from there, access the AKS worker nodes (the corresponding public key is distributed and “injected” in the AKS nodes as part of the IaC execution).   But it remains a pretty cumbersome, with a “two hops” process and this solution is not necessarily working for other Kubernetes platforms (not working in AWS EK for example).     But better to use the “container based” way…   However, there is a method that should work for any type of Kubernetes platform and does not require any specific host account password or SSH private key.   This generic solution is what we call the "container based" way because the technique used is to start a new pod with a container and exec into it to get access to the underlying host.   Requirements   We will show and explain two methods : with an open source utility called node-shell or directly using the kubectl command, but whatever the method you choose, the requirements are the same :   The KUBECONFIG file being used to access the cluster must have admin rights to the cluster. A cluster-wide admin level kube config file is required. A namespaced kube config will not work with the information provided. Only this cluster-wide admin level provides the 'root' level access on the nodes. You must also have access from the machine you're running kubectl from a network perspective either VPN, direct, or other means.   Method 1 : node-shell plugin   The first method one relies on a small utility called node-shell that lets you start a root shell in the node's host OS running. If you have been using the Lens application (great UI administration interface for Kubernetes) you are actually already using it without knowing it 😊. Indeed Lens has a feature in the Nodes view to let you open an SSH connection to the Kubernetes nodes. Behind the scene the node-shell utility is used.     The node-shell utility can be pulled and installed either in stand-alone mode, directly with 3 commands or it can be installed through krew .   Krew is a plugin manager for kubectl, it allows you to install various plugin to increase even more the capabilities of the kubectl command.   After having installed krew by following the instructions from there it is very easy to install the node-shell plugin (as well as other nice plugins) :   kubectl krew install node-shell   From here, we simply list our nodes: kubectl get nodes   For example, in azure we would see something like:   NAME STATUS ROLES AGE VERSION aks-cas-17945816-vmss000000 Ready agent 4m54s v1.23.12 aks-compute-27817762-vmss000000 Ready agent 4m30s v1.23.12 aks-stateful-16007966-vmss000000 Ready agent 4m55s v1.23.12 aks-stateless-37479317-vmss000000 Ready agent 4m58s v1.23.12 aks-system-48059844-vmss000000 Ready agent 11m v1.23.12   Then from that output, we simply target a node. For this example we'll say the node we want to connect to is   aks-cas-17945816-vmss000000 :   kubectl node-shell aks-cas-17945816-vmss000000   At this point you should see a prompt from that node, just like if you had effectively "SSHed" into the node.   spawning "nsenter-2njj8b" on "aks-cas-17945816-vmss000000" If you don't see a command prompt, try pressing enter. root@aks-cas-17945816-vmss000000:/# id uid=0(root) gid=0(root) groups=0(root),1(daemon),2(bin),3(sys),4(adm),6(disk),10(uucp),11,20(dialout),26(tape),27(sudo) root@aks-cas-17945816-vmss000000:/# ls NOTICE.txt bin boot dev etc home initrd.img initrd.img.old lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var vmlinuz vmlinuz.old root@aks-cas-17945816-vmss000000:/# Method 2 : kubectl debug   If you don’t want (or are not allowed) to install anything on your client/jump host machine in addition to kubectl, you can use a kubectl native command called kubectl debug .   This feature is provided out of the box and appears in the Kubernetes official documentation. Here is the syntax :   kubectl debug node/<node name> -it --image=<containerized OS>   All you need to provide is the name of the node that you want to connect to and the image of the container used to debug (the image must at least have the bash command).   For example in Azure , it would be something like :   [cloud-user@pdcesx03094 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION aks-cas-17945816-vmss000000 Ready agent 3m37s v1.23.12 aks-compute-27817762-vmss000000 Ready agent 3m13s v1.23.12 aks-stateful-16007966-vmss000000 Ready agent 3m38s v1.23.12 aks-stateless-37479317-vmss000000 Ready agent 3m41s v1.23.12 aks-system-48059844-vmss000000 Ready agent 10m v1.23.12 [cloud-user@pdcesx03094 ~]$ kubectl debug node/aks-cas-17945816-vmss000000 -it --image=busybox Creating debugging pod node-debugger-aks-cas-17945816-vmss000000-m9xm8 with container debugger on node aks-cas-17945816-vmss000000. If you don't see a command prompt, try pressing enter. /#   Note that in this example, we’ve been using busybox which is a minimal linux system which does not necessarily include the system debugging tools that you might need to troubleshoot an infrastructure level issue.   Instead, you could use a more complete image that would include additional system debugging tools, such as ubuntu for example, but it would pull a larger image and run a heavier container in the cluster.   This way to get SSH access to the host is the simplest and the most generic, I was able to test it with success for every supported Kubernetes platform.   Example in GCP:     Example in AWS:     Example in opensource K8s:     Example in RHEL OpenShift:     Finally, please note that there are a few important things to know about the kubectl debug command though, here is an extract from the Kubernetes documentation :     Conclusion   As we’ve seen, the two techniques ( node-shell and kubectl debug ) work, no matter what the cloud provider is, how the network has been secured or if host accounts public keys have been "cloud-init" loaded or not.   However, while required in some situation, such type of direct access should remain an exception and should be done very carefully only by the SAS or Kubernetes administrator (not every developer !) for ad hoc debugging or troubleshooting.   As rightfully noted by the SAS R&D:   “This kind SSH access could be abused to do additional node setup, but it is a bad practice since nodes are ephemeral and can come and go with autoscaling etc. Node modifications should be performed via DaemonSets or similar.   While abuse cannot be ruled out, there are still legitimate use cases for access nodes directly, to explore node configurations, to debug issues (e.g., to investigate why the Daemonset does not work as intended) etc.”   That's all, thanks for reading! ... View more

CAS (Cloud Analytic Services) is a critical component of the SAS Viya platform. It is usually the primary Analytics engine used by SAS users to run their models and crunch large volumes of data to obtain advanced analytical results.   CAS also underpins many of our products with reporting capabilities, like Visual Analytics, Model Studio, etc…. In such case many users sessions have active CAS sessions and the last thing we want is to have an interruption of our Cloud Analytics Services (which could happen if a CAS pod is terminated) and lose all the user sessions.   It is important to make sure that CAS is properly configured from a Kubernetes perspective so it can use as much resources as needed and is protected against a potential Kubernetes eviction.   In this first part of the blog, we start with a reminder on how the Pods resources are defined and managed in Kubernetes, what are the Pods "QoS" classes and then we focus on how CAS is configured in this regard.   Kubernetes resources requests and limits   As explained in the Kubernetes documentation, you can define, in the pod’s specifications, how much of CPU and memory each container needs (“request”). The kube-scheduler uses this information to decide which node to place the Pod on.   You can also specify a resource limit for a container, and the kubelet (kubernetes agent running on each node) enforces those limits so that the running container is not allowed to use more of that resource than the limit you set.   The kubelet also reserves at least the request amount of that system resource specifically for that container to use. It means that if the request is really high (and close to the node’s maximum capacity), the effective result might be the entire node reserved for just one container (in its pod).   Let’s see an example !   We can run the kubectl command (in the screenshot an alias "k" is set), to extract and display the resources definitions for all the containers of a given pod.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   We can see that for this pod, the CPU and memory requests are respectively 50 millicores (0,05 CPU) and 50Mi (Mebibytes, around 52MB) for the sas-audit container (there is only one container in this pod).   Actually, most of the SAS Viya microservices have these resource settings by default.   It is not a huge resource request, so the pod should be easily scheduled by Kubernetes, unless all available nodes resources are already "reserved".   Now, the limits are set respectively to 500 millicores (0,5 core) and 500Mi of memory. So, once the pod has been scheduled and is running on a node, the kubelet starts to monitor its resources consumption. If more CPU is used by the container, it will be throttled and if it exceeds 500MB of memory utilization, the container will be in danger of being terminated.   Note: Since the command to show the resource specifications of a pod is pretty handy, I have reproduced it below in a form that you can copy and paste, for your own benefit 😊 (just adjust the namespace and pod names).       kubectl -n <NAMESPACE> get pod <PODNAME> -o json | jq ".spec.containers[].name, .spec.containers[].resources" # extra tip:to get only the information for a given container, simply use an index # example, to only see the "sas-cas-server" container resource definitions: kubectl -n <NAMESPACE> get pod sas-cas-server-default-controller -o json | jq ".spec.containers[0].name, .spec.containers[0].resources"       Pods Quality of Service (QoS)   Now, before looking specifically at CAS, there is another important Kubernetes concept to understand: Kubernetes provides different classes of Quality of Service (QoS) to pods depending on what they request and what limits are set for them.   Kubernetes uses the Pod’s QoS class to make decisions about evicting Pods when the node's resources are exceeded. So when several pods have been scheduled on a node that is starting to run out of resources (because the pods are consuming more and more resources) Kubernetes will determine which pods to evict first based on their QoS (in order to protect the node).   The QoS class of the pod is only determined by the resources request and limit values set for its containers.   The table below summarizes the configuration and Kubernetes management of the pods for the 3 QoS classes : "best effort", "burstable" and "guaranteed".     CAS Resource settings   Now that we understand how the Kubernetes pods are configured in terms or resource requests and limits, we can look at how the CAS pods resources are configured.   Let’s review the 3 possible options* for the CAS pod resource settings: CAS auto-resourcing (default) Customized values Initial values   (*) To see how to achieve one or the other configuration option, you can refer to the SAS official documentation and to the SAS Viya 4 Architecture VLE corresponding module.   With the first option, "CAS auto-resourcing" we let the CAS Deployment operator discover the environment (and more specifically the nodes with the "CAS" label) and determine the appropriate CAS pods resource request and limit values.   With CAS resources "Customized values", we disable the CAS auto-resources transformer references in the kustomization.yaml file and use a specific transformer to set the CAS pods resource requests and limits to specific values of our choice. The CAS pods "QoS" depends on the values that are set using the chosen option.   With the last option, we let the CAS Pods start up with the setting defined in the initial resource definitions (that’s what you get when you disable the CAS auto-resourcing and don’t use a transformer to set your own values). The default request values are 250 millicores (1/4 of a core) for the CPU and 2Gi for the memory and there is no limit defined, which places the CAS pods in the “burstable” QoS class.  Note that this is "Initial values" option is purely a PoC / Dev / Test style deployment.  For production workloads you’ll want to set an appropriate values with one of the two first options.     CAS auto-resourcing is the recommended choice and is enabled if you apply the initial kustomization.yaml file provided in the official documentation.   Until recently, with CAS auto-resourcing, the values were set in a way that all containers would have the same values for request and limits, which provide Guaranteed "QoS" to CAS pods, (which is the highest Quality of service for a pod).   However, there has been some changes to this strategy and that is the topic of the upcoming second part of this post. 🙂   Find more articles from SAS Global Enablement and Learning here. ... View more

Q&A, Slides, and On-Demand Recording Watch this Ask the Expert session to learn about this topic in specific relation to open source Kubernetes.    Watch the Webinar   You will learn: Basics of Kubernetes infrastructure. Supported Kubernetes platforms for SAS Viya. Basics of the IaC tools. Main prerequisites for Viya deployments. Specifics of a Viya deployment on on-premises, open source Kubernetes.   The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.   Q&A Can SAS Viya be installed on OpenShift on AWS, and will that be supported? When considering the deployment and subsequent use of Viya on a Kubernetes platform refer to SAS’s Support statements first. There are 2 levels of support: Standard Support and Limited Support. As of February 2023, only Red Hat OpenShift Container Platform (OCP) 4.9 - 4.11 on VMware vSphere 7.0.1 or later is available with Standard Support. Red Hat OpenShift Container Platform on other platforms would be considered as Limited Support. SAS continually evolves levels of support for Viya on various platforms so we encourage interested parties to check on the supported distributions by reading the relevant SAS documentation.   Reason for my OpenShift question is our Security doesn't want to use EKS on AWS and OpenShift is the only alternative. It sounds like the question is an architecture related question where one aspect “security” is a driver when choosing which Kubernetes platform to choose. Perhaps the customer organization can speak with AWS and register their security concerns about EKS.   How can I migrate Terraform scripts to CloudFormation scripts? Translation of Terraform scripts to AWS CloudFormation scripts is beyond the scope of this presentation. SAS’s infrastructure provisioning tools (aka IaC: Azure, AWS, GCP, Opensource), are currently built using Terraform. But if you want to provision your cluster using CloudFormation, it's not an issue. SAS’s provided IaC tools help the customer to provision their cluster, but customers can also use their own tools to provision their Kubernetes cluster in the Cloud. So, using CloudFormation or the even the Cloud portal to create the Cluster infrastructure is possible.   Please explain how many node pools and how big they are in SAS Viya on Microsoft Azure Marketplace. The answer to this question is: 1 CAS node pool, 1 Compute node pool, 1 stateful node pool, 1 stateless node pool, 1 system node pool. In each node pool there is one node by default, but it can automatically scale up to 5 nodes. The CAS Server is configured for SMP. The instance types are listed in the documentation links. We will request that additional details you have raised in this question be added to the documentation or FAQs. The relevant documentation links are: SAS Viya on Microsoft Azure SAS Viya on Microsoft Azure FAQ SAS Communities and SAS Viya on Microsoft Azure   With previous SAS versions (9.x), we could go through the EEC team for environment sizing recommendations. With Kubernetes and the new Viya, how should we approach environment sizing? You can still contact the SAS World Wide Sizing Team through the local SAS office to get help with appropriate recommendations to meet the need of your workload in SAS Viya. In addition to or instead of using the World Wide Sizing team, customer organizations may want to consider an iterative approach with Viya to determine the right and type of infrastructure for their specific use cases. One of the benefits of using Kubernetes in a Cloud Platform like Azure, GCP and AWS is the speed and flexibility at which the required infrastructure can be built, modified, and deleted/removed using infrastructure as code (aka IaC). For some customers they may wish to start on the small side of the estimate and then grow/expand the infrastructure to meet the demands of the user community over time. So, combining the results from SAS World Wide Sizing and iterative evaluations will allow customers to identify what infrastructure is required for their needs.   Regarding node pools, for example stateless node pool assume there are 3 nodes, do we install all the stateless application in one pod in one node and the same will be repeated for all the 3 nodes? Or how does this node pooling actually work? Your question relates to the placement of pods. This falls under the category of a Kubernetes concept termed “workload placement”. We recommend you watch these series of short YouTube videos “SAS Viya Topologies” to learn more:  https://www.youtube.com/results?search_query=viya+topolgies   Does SAS get a lot of pushbacks from customers who don't use Kubernetes or don't have Kubernetes resources? SAS provides various options for customers who want Viya 4 but may not be ready to host or manage themselves, or generally not in a position to move to Viya 4. This includes using SAS Cloud (SAS hosted) and Remote Managed Services provided by SAS. In addition, customers could work with technical partners to provide a hosted environment. Customers typically explain their modernization/roadmaps with SAS and then a plan can be put in place for an agreed approach going forward.   Is RHEL also possible as underlying Linux OS, instead of Ubuntu? If the customer is thinking of deploying upstream opensource Kubernetes and is happy to use their own provisioning methods/code, they can choose a flavour & version of the Linux operating system that can meet the requirements of upstream opensource Kubernetes then they can proceed. The requirements for SAS Viya Platform and Upstream Source Kubernetes is documented here. If a customer prefers to use SAS’s Infrastructure as Code tools to deploy and Upstream Open Source Kubernetes cluster they will be required to use Ubuntu Linux LTS 20.04 or 22.04 (correct at March 2 nd , 2023).   How do you implement 60 tenants in Viya 4? The SAS Viya platform can be deployed with multi-tenancy enabled. The use of multitenancy can be for several use cases. Without knowing any specific details for your use case(s) we recommend you speaking a SAS technical representative or SAS approved partner in your area to discuss what options are available for your use case. You can also try asking a more detailed version of your question on SAS Communities. Current known limitations to Multi-tenancy are listed here.   Recommended Resources SAS Viya Blog Collection on SAS Communities Running SAS Viya in Upstream Open Source Kubernetes – part 1 Running SAS Viya in Upstream Open source Kubernetes – part 2 Running SAS Viya in Upstream Open source Kubernetes – part 3 Moving from SAS®9 to SAS® Viya® Please see additional resources in the attached slide deck.   Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars. ... View more

My apologies but it looks like something went wrong with the email announcement of my latest article. Here is the correct link: New SAS Viya Deployment Methods Thanks ! ... View more

Sometime ago, I wrote a blog to introduce and compare the Viya 4 deployment methods, called “All roads lead to Rome : Three Tables and one Diagram to review the SAS Viya 4 deployment methods”   As we begin 2023, and several things have been changing in this area, I thought it was a good time to revisit the Viya 4 deployment methods landscape.   After a small recap of the existing methods, we will present the new "sas-orchestration command" deployment method, explain recent significant changes in the DaC (Deployment as Code) project and even compare the 4 official ways to deploy SAS Viya.   Recap Let’s start with a little recap on the SAS Viya deployment methods. At the beginning we only had the so-called “manual” deployment method, where you were simply applying to the cluster the Kubernetes manifest that was generated by the Kustomize build (the famous “site.yaml” file), by running the kubectl apply commands.   …Then the Deployment Operator method was introduced. It leverages the Kubernetes Operator concept where you can create your own custom resource definitions in Kubernetes.   So we first need to install the SAS Deployment Operator and it will then manage and watch any change happening in the "SASDeployment" Custom Resources (CR) that were applied in the cluster.   Each "SASDeployment" CR corresponds to a distinct SAS Viya Deployment within the same cluster.   The Deployment Operator monitors the Custom Resource specification and automatically pulls the corresponding deployment assets (sas-base folder) then run the kustomize build and kubectl apply commands for you.   Once you are familiar with it, it really simplifies the deployment and update steps.   However there is no magic, even with the Deployment Operator method you still need to write the kustomization.yaml and various kustomize override yaml files (placed in the “site-config” folder) by yourself…   It would be so nice if we could avoid to do that…but wait a minute…actually there is one deployment method that saves you from these tedious, error-prone and always changing tasks!   Here comes the 3 rd method : The SAS Viya 4 deployment GitHub project (aka "DaC" for "Deployment as Code"). Let’s explain a little bit more what it is.   Deployment as Code ( DaC)   This method relies on a project that is maintained in one of the sassoftware GitHub repository which is simply called “The SAS Viya 4 deployment tool”.   The SAS Viya 4 deployment tool contains Ansible code that :   Creates the cluster "baseline" in an existing Kubernetes environment for use with SAS Viya 4, Install all the pre-requisites (Ingress NGINX Controller, NFS provisioner, cert-manager, metrics server,…) Generates the manifest for a SAS Viya software order, It can automatically implement several common customizations (number of CAS workers, mirror, etc…), depending on the values given in the ansible vars file (see the CONFIG-VARS page for all the capabilities). and then deploys that order into the specified Kubernetes environment.   One of the major limitations, that one needs to know though, is that the DaC viya4-deployment cannot be used to perform SAS Viya versions updates.   If you have an existing SAS Viya environment that you initially installed with the viya4-deployment project, you’ll need to follow the instructions in "Updating Software" in the SAS Viya Operations Guide where you apply the standard re-deployment methods and you are also expected to modify the documented steps to accommodate the slightly different directory structure.   So, what’s new ?   If you are familiar with SAS Viya deployments, you were probably already aware of the 3 methods presented above.   So what is changing, with the new SAS Viya Stable version 2022.12 ?   Firstly, there is now a 4th deployment method. And secondly, the DaC method is also being changed.   A fourth method   The new method (introduced in the Viya 2022.12 stable version) is called "sas-orchestration command" because it relies on the same sas-orchestration Docker image that we already use with the Deployment Operator method to create the Deployment CR.   But this time, instead of using the "create" function of the sas-orchestration container, we use the "deploy" function which successively pulls the deployment assets, build the manifest (with kustomize ) and apply it (with kubectl apply commands).   With this fourth method, while we perform an "orchestrated" deployment, we don't need to deploy the SAS Deployment Operator into the Kubernetes cluster. The deployment is managed from  "outside" of the cluster, whereas with the Deployment Operator, the deployment is done by the Operator, inside the cluster.   The main reason to offer this alternative method was to remove the need to create a cluster wide Operator with the associated Cluster privileges (which is not acceptable to all customers).   In terms of utilization, you can run it from any machine (with Docker installed) from where you can connect to the Kubernetes cluster API as a Kubernetes administrator with cluster permissions.   You need to download the license, certificate files and deployment assets, then pull the corresponding sas-orchestration Docker image, and from there you can run the docker run command and call the deploy function to deploy or update your SAS Viya deployment to any specific version or cadence (as long as it is a supported version).   Here is the syntax :   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     And a simple example :     When you run the deploy command (and assuming your configuration is correct), it will output the executed steps, for example :     We can see that a lot of things are happening behind the scenes (artifacts generation, kustomization and manifest generation, version check, pre-deployment steps, kubectl apply, post deployment steps, etc…)   At the end, we should see a “success” message.       To summarize, this fourth method reduces the level of complexity by removing the need to go through many of the individual commands that an administrator would see in a manual deployment. It also embeds various checks and controls, for example to prevent you from doing something that would not be supported in terms of versions update.     New DaC release with SAS deployment operator and orchestration commands support   While it does not change anything from a usability perspective, the underlying Viya deployment method used by the DaC has changed.   Until very recently, the method used by the DaC to deploy Viya was the manual method (with the kustomize and kubectl apply commands).   However since the release of stable 2022.12, the DaC has been updated to use either the SAS Deployment Operator or the new “sas-orchestration command” method (described above).   Several new variables, such as V4_DEPLOYMENT_OPERATOR_ENABLED , V4_DEPLOYMENT_OPERATOR_SCOPE ,   V4_DEPLOYMENT_OPERATOR_NAMESPACE , V4_DEPLOYMENT_OPERATOR_CRB  have been introduced to allow the DaC user to choose which of the two methods to use and how to configure them.   When not specified explicitly in the ansible vars file, the default for the DaC is to deploy the Deployment Operator with a "cluster" scope (in its own "sasoperator" namespace).   This change means that from now, the DaC always uses the sas-orchestration tooling, which introduces a new requirement : you need to have Docker installed on the machine from where you run the DaC tool and the executing user should be able to access it.   Comparing the methods       Choosing the right method is one of the key decisions that needs to be made during the Architecture design phase of the Viya implementation project.   The choice, not only impacts the way the initial deployment is performed, but also the way the future update operations will be done and how the customer’s specific configuration changes will be made…   The tables below can be used as a starting point of your conversation with the customer.           Benefits, Capabilities, Restrictions and use cases   Method Benefits and Capabilities Restrictions, additional considerations Target use case Deployment Operator · Automates deployment steps (pull assets, kustomize, kubectl apply)   · During updates, automates the pull of newer deployment assets and several Deployment notes steps   · Manage the deployment / update of multiple Viya deployments of the same version within the same cluster   · Feature to schedule daily  automated release updates (within the same version)   · Validation of adherence to the Viya policy (e.g.: Version update path controls)   · Requires service account to run the operator with cluster-wide RBAC permissions in the cluster   · Requires docker on the jump host to generate the SAS Deployment CR   · No capability to apply resources based on permissions level (cluster-wide, namespace-scoped, etc...)   · Takes longer to apply configuration changes. For most (all ?) configuration changes - like changing the number of CAS workers - the reconcile operation redeploy, restart postgres instances.   · Increased complexity of configuring a private registry for the Viya images.   · Use in offline/dark sites (no internet access) requires extra httpd server with a mirror of the entitlements   · Operator driven Viya deployment happening within the cluster (it might meet the customer preferences or requirements). · SAS-Supplied automation (“Black box*” deployment). sas-orchestration commands · Automates deployment steps (pull assets, kustomize, kubectl apply)   · During updates, automates the pull of newer deployment assets and several Deployment notes steps.   · Does not require a service account to run the operator with cluster-wide RBAC   · Validation of adherence to the Viya policy (e.g.: Version update path controls)   ·The sas-orchestration image includes a supported version of kubectl and Kustomize   · Requires docker on the jump host to deploy SAS Viya   · No capability to apply resources based on permissions level (cluster-wide, namespace-scoped, etc...)   · Takes longer to apply configuration changes. · Container-driven Viya deployment · A Docker image, “sas-orchestration”, is used to automate the deployment process from outside the cluster. · SAS-Supplied automation. Kubectl apply commands (“manual”) · Better control of the process   · Kustomize-generated manifest file is available for troubleshooting and further processing (for example security scans).   · The most flexible method when integrating with customer automation tooling   · Declarative - The kubernetes way   · No SAS-supplied automation features   · No validation of adherence to Viya Policy · Full control is required · Customers who want to use standard native Kubernetes deployment process (kubectl commands) · Customers with strict requirements on Kubernetes permissions with distinct roles to perform deployment and configuration operations. SAS Viya 4 Deployment GitHub project · Automates everything (initial config, pull assets, kustomize, kubectl apply)   · Automates several common customizations documented in component’s specific READMEs   ·The source code is open and can be customized/changed if needed   · Can use either the Deployment Operator Method or the sas-orchestration commands   · Can automate the pre-requisite installation (ingress nginx, nfs provisioner, etc…)   · Can automate Viya monitoring tools deployments · Same considerations as “Deployment Operator” or “sas-orchestration” command methods   · Specific way to change the configuration, with adjustments   · Does not perform updates (documented steps must be followed with adjustments regarding the directory structure)   · Not supported for Red Hat OpenShift   · Issues with the Deployment method fell under the sassoftware GitHub support policy (instead of standard support) · Simplified initial deployments, with Maximum automation (PoC, tests) · Combination/ Integration with the IaC · All target use cases  listed for “Deployment operator” and “sas-orchestration commands” methods · SAS-provided open-source deployment automation     Overall Deployment and Update steps   Standard steps for a new Deployment (with no multi-tenancy, no SingleStore, no mirror registry):   Manual DepOp Orchestration command DAC 1. Install all the Viya pre-requisites or ensure they are in place.   2. Pull the Deployment assets   3. Prepare the Viya  Kustomization.yaml and site-config custom configuration   4. Build the manifest with Kustomize   5. Apply the manifest (4 commands to apply per type of Kubernetes objects) 1.  Install all the Viya pre-requisites or ensure they are in place.   2. Pull the Deployment assets   3.  Configure and Install the Deployment Operator   4.  Prepare the Viya Kustomization.yaml and site-config custom configuration   5. Pull the orchestration CLI image   6.  Generate the SAS Deployment CR   7.  Apply the SAS Deployment CR 1. Install all the Viya pre-requisites or ensure they are in place.   2. Pull the Deployment assets   3. Prepare the Viya  Kustomization.yaml and site-config custom configuration   4. Pull the orchestration CLI image   5.  Run the orchestration “deploy” command   1. Pull the DaC git repository   2. Configure the deployment in the ansible vars.yaml   3. Run the playbook/docker command to install the pre-requisites and deploy Viya   Standard steps for a version update (with no multi-tenancy, no SingleStore, no mirror registry):   Manual DepOp Orchestration command DAC Prepare for the update:   1. Create a Backup   2. Review System Requirements and Resources   3. Download New Deployment Assets   4. Read the Deployment Notes and apply “before deployment commands”   5. Perform Required Tasks by Component   Deploy the new version   6. Re-build the manifest with Kustomize   7. Apply the manifest (4 commands to apply per type of Kubernetes objects)   Apply “After Deployment commands” in Deployment notes   Prepare for the update:   1. Create a Backup   2. Review System Requirements and Resources   3. Optional : update the Deployment Operator   4. Download New Files from My SAS   5. Read the Deployment Notes and apply "before deployment commands"*   Apply the SASDeployment Custom Resource   6. Update the orchestration image.   7. Create the SASDeployment custom resource with updated values.   8. Apply the SASDeployment CR   Apply “After Deployment commands” in Deployment notes*   Prepare for the update:   1. Create a Backup   2. Review System Requirements and Resources   3. Download New Files from My SAS   4. Read the Deployment Notes and apply "before deployment commands"*   Deploy the new version   5. Update the orchestration image (must match the version defined in the deployment assets README).   6. Run the sas-orchestration deploy command with updated version value   Apply “After Deployment commands” in Deployment notes*     Depending on the underlying method (Deployment operator or sas-orchestration), follow the documented steps and make the correct adjustments as the files organization, Kustomization.yaml form and ways to redeploy with the updated version will be different.   (*) When using the deployment operator or the sas-orchestration command, some deployment notes are not required.   Overall Summary   Manual DepOp Orchestration command DAC · Standard Kubernetes deployment with kubectl apply command, more control but no SAS Automation. · SAS automated /orchestrated deployment with an Operator.   · Follows the K8s Operator pattern but more Complex and many steps.   · Reduces the number of steps in the update process. · SAS automated/orchestrated  deployment, without the need of a full blown Operator. · Ansible driven method with a variable files. Simplifies the pre-requisites setup, common configurations and deployment for initial deployments.   · But version updates and non-standard configuration requires adjustments.    These methods are using the DaaS (for "Deployment as a Service") orchestration tooling which is aimed to embed additional automation and product specific configurations and checks for SAS Viya deployments in the future.   Changing method Changing from one deployment method to another is possible but requires an outage and a redeployment of the software and possibly a migration of content.   Moving away from the Deployment Operator is the operation that requires the most amount of work (full redeployment with backup and restore operations).   The possible scenarios are officially documented here.   Conclusion   The key takeaway of this post is that there were 2 significant changes made to the deployment methods with the last 2022 SAS Viya version (stable 2022.12): A fourth deployment method "sas-orchestration command" has been introduced. The DaC tool now relies on the Orchestrated deployment methods (instead of the “manual” apply commands). The deployment method choice is important as it determines how to manage your Viya platform in the long term.   It is something that must be discussed and agreed with the customer as part of the Architecture design and documented in the D30 (or equivalent Architecture Definition document).   Hopefully this post will be helpful to achieve this objective.   Thanks for reading !   Find more articles from SAS Global Enablement and Learning here. ... View more

In this post, we'll continue our journey toward a SAS Viya Deployment in a an "on-premise" Open Source Kubernetes cluster.   The first post discussed the scope of support, the platform requirements (calico, containerd, kube-vip, etc…) and introduced the SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes project for Opensource Kubernetes.   The second one focused on the IaC tool, its specific requirements and how to configure and run it.   But if your customer has chosen to deploy Viya on-premise using an opensource Kubernetes and you are using the SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes tool to prepare the cluster, you will very likely need to look under the hood, and understand how the tool is working to be in position to efficiently troubleshoot any issue.   So, in this 3rd post, we’ll give you some tips to troubleshoot the IaC tool for Opensource Kubernetes and also share some experience and challenges we faced while using the tool.   Troubleshoot the IaC tool for Open source Kubernetes   Once you have your Ansible inventory and vars file ready (as discussed in the previous post), all you have to do is to run the oss-k8s.sh script that encapsulates the ansible playbooks execution. This page in the viya4-iac-k8s project repository explains how the script can be used.   As explained in the first post, you can call the oss-k8s.sh in two ways: one to prepare the system (" oss-k8s.sh setup "), and one to install the Kubernetes software (" oss-k8s.sh install ").   In case the first the "setup" command fails, for some reason, you can fix the root cause and try again running the same command (it corresponds to what we call a "fully idempotent" playbook). However, for the "install" command there is an "uninstall" command that you need to run when you want to make a second attempt at running the Kubernetes installation. The "uninstall" command takes care of the required clean-up that needs to take place before a new try.     Troubleshoot with Ansible   The " oss-k8s.sh setup " and " oss-k8s.sh install " commands correspond to 2 distinct Ansible playbooks: playbooks/systems-install.yaml and playbooks/kubernetes-install.yaml In case of failure, you might want to bypass the oss-k8s.sh script to run the playbook by yourself …the only thing is that you need to set the required variables.   See below an example on how to do that:   ANSIBLE_USER=cloud-user ANSIBLE_PASSWORD=lnxsas BASEDIR="/home/cloud-user/viya4-iac-k8s" SYSTEM="bare_metal" ANSIBLE_INVENTORY="inventory" ANSIBLE_VARS="ansible-vars.yaml" IAC_TOOLING="terraform" # default value - although we don't use TF for Baremetal as the opposite to "docker" # install pre-requisites ansible-galaxy collection install -r "$BASEDIR/requirements.yaml" -f cd $BASEDIR # "oss-k8.sh setup" step by step ansible-playbook -i $ANSIBLE_INVENTORY --extra-vars "deployment_type=$SYSTEM" --extra-vars @$ANSIBLE_VARS playbooks/systems-install.yaml --flush-cache --tags install --step # "oss-k8.sh install" from a given task ansible-playbook -i $ANSIBLE_INVENTORY --extra-vars "deployment_type=$SYSTEM" --extra-vars "iac_tooling=$IAC_TOOLING" --extra-vars "iac_inventory_dir=$BASEDIR" --extra-vars "k8s_tool_base"=$BASEDIR --extra-vars @$ANSIBLE_VARS $BASEDIR/playbooks/kubernetes-install.yaml --flush-cache --tags install --start-at-task "Register Git HASH information" Once you know how to run the tool from the ansible command line, you can use all use your favorite ansible "troubleshooting tricks"!   Such as increasing the ansible log verbosity ( -vvvv ), run the task one by one (with “ --step ”) or recover from a failure by running it from a specific checkpoint (“ --start-at-task ”).   Finally, if just like me, you are struggling to read ansible error message when displayed during a playbook failure, like in the screenshot below:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   then you can use the trick that I’m sharing here (that is not something new but remains pretty useful when it comes to ansible troubleshooting).   If you have Notepad++, just copy/paste the ansible output and perform a global replacement as shown below (don’t forget to click on "Extended").   And it should you the error message in a much more readable form. 😊   Troubleshoot a "Kubeadm init" failure.   The bulk of the work performed by the IaC tool is the installation of the open-source Kubernetes software itself on our nodes. Kubeadm (which is one of the official tools referenced in the opensource K8s project) is used for that purpose and it happens in the " kubeadm init " task.   If the task fails, then you will probably have an error message that tells you to look at the kubelet log or in the containers logs. The kubelet is an agent that runs on each node of a Kubernetes cluster to perform various tasks (register the node, monitor the pods resource utilization, etc…).   So, you’ll have to jump on your control plane machine (there can be one or three) and inspect the kubelet service.   There are a few ways to see the kubelet service logs:   sudo systemctl status kubelet -l sudo journalctl -xeu kubelet | less   You might also need to look in the containers logs that are stored in /var/log/pods/<pod ID> . For example, the API Server log can be found in /var/log/pods/<api_server pod ID>/kube-apiserver   If the Kubernetes API server has started (in spite of the task failure and many error messages in the kubelet logs) you could also try to connect to your cluster.   The KUBECONFIG file would not have been generated yet, but you can find one on the one of the master node ( /etc/kubernetes/admin.conf ) and use it to get access to the cluster (for example with Lens or the kubectl command) and assess the issue.   If you need to run some commands to inspect the running containers or associated images, you can use the crictl tool (which is installed by the IaC and offers a long list of commands to interact with the containers).   sudo crictl ps   The network configuration needs to define exactly as expected for the "kubeadm init" task to be successful.   So, you might want to make sure that the floating IP that was defined during the IaC configuration ( kubernetes_vip_ip ) has been attached to your network interface and also ensure that the DNS name that you have associated to it ( kubernetes_vip_loadbalanced_dns ) is working.   # jump on one of the control plane node ssh <control plane node> # show the IP addresses attached to the network interface sudo ip addr show eth0 # ensure the DNS name (kubernetes_vip_loadbalanced_dns) # is associated to the floating IP (kubernetes_vip_ip) nslookup osk-api.viya.mycompany.com   First experience feedback   While working on new hands-on of the Viya 4 Deployment workshop VLE (available soon 😊), to provide the opportunity to SAS employees to try out this new IaC tool (and setup their own opensource Kubernetes cluster on a bunch of RACE machines), I faced several challenges issues and learned a few things. I’ll share some of them here 😊   Topology traps As we’ve seen, if you are ok to deploy a less available and scalable K8s cluster, you can really reduce the number of required machines. Some "non-kubernetes" roles such as "jumphost" or "NFS" can also be placed on hosts playing the role of Kubernetes nodes, they don’t need to be on dedicated machines.   However, be careful, there might be some incompatible combinations!   As an example, you can’t place jumphost and NFS roles on the same node (it would trigger problems with the NFS share mount that is configured on the jumphost).   The reboot issue   During the playbook run, there is an ansible task to reboot all the nodes defined in the inventory, in order to force the OS to pick up some low-level system changes (like the GRUB configuration change).   But if you were using the "jumphost" provisioned machine as your ansible controller, then you get automatically disconnected in the middle of the playbook execution!!! (Since ansible forces the machine from where your run the playbook to reboot). And obviously, the rest of the playbook tasks cannot be executed.   One way to avoid this trap is to run the ansible playbook from another external machine (outside of the hosts defined in the Ansible inventory) and, as such, is not affected by the "reboot all nodes" task.   The two scenarios are represented in the schema below.     However, it means that we need yet another machine in addition of the ones we have define in the inventory...   There is a trick to avoid this additional machine requirement, though in this kind of topology.   The reboot is only really required on the Kubernetes node, so they can pick up the cgroup directives defined in the GRUB configuration. A reboot is not required for the Jump Host machine.   So, to avoid the need of another external ansible controller, we can just slightly modify the playbook to make the "reboot" task only happen on the Kubernetes nodes 😊 (instead of all the nodes).   Here’s an example on how to automatically modify the original playbook, for that:   # Override the kubernetes-install.yaml playbook to only reboot K8s nodes sed -i '/- hosts: all/c\- hosts: k8s' ~/viya4-iac-k8s/playbooks/kubernetes-install.yaml   This 3rd scenario is represented below:     Note that this issue was reported here   Kubeadm requirements The IaC tool relies on Kubeadm to install Kubernetes.   Kubeadm is defined in the official Kubernetes documentation as "a tool built to provide (…) fast paths for creating Kubernetes clusters".   Kubeadm also has its own requirements and performs a series of “pre-flight” checks and will fail if any of the requirement is not fulfilled.     The Kubeadm requirements are not always explicitly repeated in the IaC documentation. As an example, the system swap is expected to be disabled on the linux hosts. If it is not the case, then the IaC tool fails (this issue has been reported here).   If you want an automated way to disable the swap on your Kubernetes nodes before running the IaC tool, you can use the snippet below (double check your indentation if you copy the code from here).   bash -c "cat << EOF > ~/viya4-iac-k8s/disable-swap.yaml --- - hosts: k8s tasks: - name: Disable SWAP since kubernetes can't work with swap enabled (1/2) shell: | swapoff -a - name: Disable SWAP in fstab since kubernetes can't work with swap enabled (2/2) replace: path: /etc/fstab regexp: '^([^#].*?\sswap\s+sw\s+.*)$' replace: '# \1' EOF" cd ~/viya4-iac-k8s ansible-playbook -i inventory disable-swap.yaml -b   If you face an issue with the IaC tool, you should do the following:   Try to troubleshoot the issue by yourself Check in the repository issue list to see if it is a known issue If you are still stuck, create a new GitHub issue in the IaC repository   Conclusion   As you may already have realized (especially if you’ve read the whole post series), deploying your own Kubernetes cluster (even with the IAC tool) might not be so easy and seamless.   Given the complexity of the infrastructure requirements (especially in terms of network and storage), the inherent sophistication of Kubernetes itself, and the variety of environment’s infrastructures it is not really such a big surprise…   That is mainly why we should strongly encourage organizations choosing the "on-prem" Open-source Kubernetes platform for Viya, to ensure they have the Kubernetes and container technologies knowledge and skills "in house".   Having a good experience and knowledge of the linux system and Ansible will also be of a great help in order to provision the cluster with the IaC tool and hopefully this post also provides some useful tips for a successful implementation.   In our next and last post of the series, we’ll finally talk about the SAS Viya software deployment itself with a zoom on the SAS Viya 4 Deployment GitHub project (also known as “DaC” for Deployment as Code).   Thanks for reading and for your comments!   Find more articles from SAS Global Enablement and Learning here. ... View more

Since June 2022 (stable 2022.1.2), customers can deploy SAS Viya in their own "on-premise" Opensource Kubernetes cluster and benefit from the standard SAS support. Using the SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes they can automate the provisioning of their Kubernetes cluster.   In this second part of the “Viya in Open source Kubernetes blog”, we want to review the requirements for the IaC; and see how to configure and execute it for a “bare-metal” deployment.   IaC for Open Source Kubernetes requirements   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The IaC tool is provided by SAS as a GitHub project (sassoftware/viya4-iac-k8s) to automate the installation of an open-source Kubernetes cluster either in a vSphere environment or on a bunch of "bare-os" machines. But it brings its own specific requirements which are not to be confused with the general "Viya 4 on Upstream Open Source Kubernetes platform" requirements themselves.   The use of the IaC tool is NOT mandatory. A customer organization can provide an "on-prem" Kubernetes cluster by its own means. If the environment complies to the documented Upstream Open Source platform requirements such as Calico, containerd, etc…then that’s enough for this customer to benefit from the standard support.   Now, if the customer wants to make their life easier and use the SAS provided IaC tool ( viya4-iac-K8s ) to automate the Kubernetes installation and ensure that they build a SAS supported open-source Kubernetes environment, there are some extra requirements !   As an example, the IaC tool currently only supports Ubuntu OS based host machines. However a customer who would deploy the upstream open-source K8s using their own infrastructure tooling could use an alternative linux OS for their host machines.   Main requirements (OS, tooling, etc…) The "SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes", as of today is only supported on host machines running Ubuntu Linux LTS 20.04. (work is currently in progress to add the support for Ubuntu LTS 22.04)   In addition, the machines need to have a default user account with password-less sudo capabilities. If you have some experience with Viya 3.5 or Ansible in general, this requirement will be familiar to you.   To run the IaC script you also need some tools (Ansible, terraform, docker). As described in the table below (from the first post), the exact requirements depend on the Deployment method (bash or docker) and deployment type ("bare-metal" vs "vSphere").     Another important requirement is that all the machines in the collection must have the same date and time. While not explicitly listed in the project repository this requirement is very important... if one of your node is 2 minutes behind the others, the installation will not succeed. When trying to connect to your API server you’ll see this kind of error message :             Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-09-05T09:11:23Z is before 2022-09-05T09:12:15Z             It might seem obvious that all servers are at the same time, however in a VMWare environment, on boot the VM picks up the time from the BIOS (which would be from the VMWare hosts) and real life experience taught us, that the time could drift on specific VMWare hosts...resulting in different date/time on the collection’s machines.   Topology   In terms of required machines, the project documentation also has the following requirements :   If you read these requirements as they are and do the maths, it adds up to a minimum of 11 machines (12 if you are planning to use an external PostgreSQL server on a dedicated machine).   Here is diagram describing this kind of topology with a 4 nodes CAS MPP and 2 compute nodes.   Upstream open-source topology (full)   That’s a lot of machines...so it might be disconcerting to a customer who is just looking at building a small test or POC environment.   Actually there are ways to successfully stand up the Kubernetes cluster with a smaller number of machines.   First, the 3 machines for the Control Plane are only required if you want an HA configuration of the Kubernetes Cluster. If you are planning to build a development, disposable test or POC environment with the IaC tool, choosing 1 machine for your Control Plane is quite sufficient.   Then, it is possible to assign multiple “roles” to the same machines. However in this case you’ll need to be careful, depending on the topology that you define, the tool could fail or you could end up with a deployment where pods can’t be scheduled.   Here is an example of an alternative topology which has a smaller footprint (with the associated diagram).   sasnode01 : jumphost sasnode02 : K8s Control Plane sasnode03 : K8s node for System + NFS sasnode04 : K8s node for CAS (SMP) sasnode05 : K8s node for Compute sasnode06 : K8s node for Stateful sasnode07 : K8s node for Stateless Upstream open-source topology (small)   Such an optimization of the number of host machines is not recommended for a production environment where a proper design and topology must be established. For example co-locating the NFS server on a K8s node is generally not a good idea as they might be in conflict for the storage resources.   Network   The main network requirements are listed in the IaC README file:     The first two items are very standard : we need our target machines (Cluster nodes, jumphost, NFS server and external PostgreSQL) to be on the same “routable” network and each machine needs to be able to talk to the others using a fix IP address.   The last item for the “floating IP addresses” is a little bit more unusual so let’s take a step back and explain why these requirements are there 🙂   When you deploy Kubernetes in the Public Cloud, the required load-balancer components are automatically and dynamically provisioned by the Cloud provider with the associated external IP addresses. Those external IP addresses can then be used to connect to the Kubernetes API or to access to the applications running inside the cluster.   In our "on-prem" bare-metal deployment (where we cannot leverage this kind of Public Cloud integration), we use kube-vip and a pool of pre-defined Virtual IP addresses (also known as “Floating” IP addresses) to provide the same capability.   The VIP addresses act just like the load balancer components a Cloud provider has in place when you are requesting an external IP for your LoadBalancer service.   In the IaC ansible "vars" file, we must provide information about the Virtual IP addresses, that is used for:   The Kubernetes Control Plane access (whether there is one or 3 CP nodes) The Ingress Controller (NGINX) load-balancer service acting as the entry point for external HTTP/s request to access the web apps and services running in the cluster. Other load-balancers services that could be required (for CAS or SAS/CONNECT access on non-HTTP/s ports)   At the end, the environment will look like in this diagram.   Understanding the IaC tool   Welcome back Ansible !   The good news is that the tool is based on Ansible and Ansible is not something completely new to us 😊.   Ansible has been around for long time and is a very popular tool to automate computers configuration and management. But Ansible is also what SAS consultants have been using for years to deploy the previous version of the Viya platform (Viya 3.5).   If you know Ansible, then you know that it works with 2 things : the "inventory" file and the "vars" file.   You can find samples of both files in the viya4-iac-k8s repository, but basically the inventory file is where you implement the topology and the vars file is where you provide the configuration values.   Inventory file   The schema below shows how to implement the “small” topology discussed before in the inventory file (with 7 machines).     Note that the repository provided sample assumes that your deployment is using an external PostgreSQL server(s) for the SAS infrastructure Data Server.   So, if you want to deploy Viya with the OOTB internal “sas-crunchy” PostgreSQL server, then you won’t need the corresponding “postgres” host groups and you must comment them in your inventory file (as in the screenshot above).   Note that a GitHub issue has been opened to request an inventory sample with an internal PostgreSQL server configuration.   Ansible vars file   A sample file is also provided for the ansible-vars.yaml .     In addition, the CONFIG-VARS page of the repository contains the variables description and default values.   Most of the variables are easy to understand and fulfill.   However there are some variables that needs a little bit more effort and thinking…One of the trickiest part of the tool configuration is probably to find the values for the following variables.     The variables values are used to meet the network requirements described above. The kubernetes_vip_ip is an IP address for the entry point of the Kubernetes cluster itself. As part of the deployment, kube-vip creates a load-balancer component with the “user provided” IP addresses to balance K8s API requests across the Control plane nodes.   The kubernetes_vip_cloud_provider_range is the range of IP's that can be assigned to new LoadBalancer services created inside your cluster. The range value is not expressed with a CIDR notation, but with the "firstIP-lastIP" format, for example : "10.96.18.5-10.96.18.8". Typically a range of 3 or 4 IP addresses should be enough : one for the web application, one for CAS, one for CONNECT, one for a specific service).   The kubernetes_vip_interface is the machine's network interface corresponding to our Virtual IP address (something like "eth0" or "ens32"). But note that in the latest IaC version, this parameter has become optional.   The kubernetes_vip_loadbalanced_dns is the DNS alias that corresponds to the IP address used to access the K8s Cluster. For an “on-prem” deployment, the corresponding A and CNAMES entries are typically  created in the corporate DNS. So how do you find the value to use for the VIP_IP and the VIP_CLOUD_PROVIDER_RANGE ?   The value is "user defined" and should not be part of any DHCP range or be assigned to any physical machine. Typically, you will ask your customer to provide a set of IP addresses on the VLAN that are not already associated to any machines and use the values of this IP addresses in your ansible-vars.yaml file.   As an example, in our RACE hands-on, we search for IP addresses that are available outside of the reserved range of IP addresses used for the RACE.EXNET hosts.   IMPORTANT: The DNS name used for the kubernetes_vip_cloud_provider_range value must exist and resolve to the IP address used for the kubernetes_vip_ip vaue before you run the “oss-k8.sh install” command   In addition, as explained in the first part of my post, for the Calico setup, we also need specify distinct CIDR IP ranges for pods and services.     But as you can see in the CONFIG-VARS page there are default values that you can use for them (without having anything specific to do) or adapt depending on the customer specific requirements.   Conclusion   It is already a pretty long and complex post with a lot of technical concepts to digest 😊, so I’ll stop here for today.   In our 3 rd and last post of this series, we’ll go a little bit more in the technical details, discuss how to troubleshoot the IaC issues using ansible, but we’ll also have a look at what’s going on during the Kubernetes software installation, then we’ll share some first “hands-on” experience feedback from our utilization of the tool.     ... View more

The June 2022 stable release of SAS Viya (2022.1.2) brought the support of upstream open source Kubernetes. This adds a second and very significant option for on-premises deployment of the Viya platform (in addition to Red Hat Openshift on vSphere). Customers can choose among many ways to install K8s, many available K8s distributions (Rancher, Mirantis, etc…), many manual or automated deployment methods (kubeadm, kops, kubespray, etc). In the first part of this blog, we’ll explain what this “upstream” term means and discuss what can indeed be considered as supported open source Kubernetes environments for SAS Viya. We will also present the new associated IaC (Infrastructure as Code) project and explain how it is organized and can be used to provision a supported Kubernetes environment.   What is supported and what is not supported   The scope of support Before reading on please familiarize yourself with SAS Technical Support statements in regards to Viya. First of all, we need to make the distinction between the "Standard” support"  - provided by the SAS Technical Support and R&D teams for fully tested and documented Kubernetes implementations, such as Azure AKS (and others listed here) - and the "Limited" support policy that is applicable for Alternative Kubernetes Distributions.   In order for your Viya deployment to benefit from the “Standard” support in an open source Kubernetes implementation, there are some strict requirements that correspond to:   “selective” configuration choices in the Kubernetes setup. the way the open-source Kubernetes project is installed and managed.   In Kubernetes implementation there are really a lot of possibilities in terms of network, storage and other infrastructure components. The "selective" configuration choices corresponds to fully tested environments and we will discuss them with more details in the next section.   In addition, SAS only support installations of the official “upstream” Open-Source K8s repository code (https://github.com/kubernetes/kubernetes), and the cluster must be created with one of the official documented ways from the kubernetes.io web page.   Many "packaged" distributions, even if they claim to be Kubernetes CNCF Certified, and even if they rely ultimately on the same upstream official project, are bringing their own “sauce” and features and as such are considered as “downstream” open source Kubernetes implementations.   Consequently, SAS does not provide "Standard" support for any kind of vendor-packaged solutions of open source Kubernetes (at least with the current stable release of SAS Viya).   However manually installing “upstream” Open source Kubernetes in production mode (with a highly available Control Plane for example), using one of the officially supported tools, is not something which is very easy to do...You might have heard about the “Kubernetes the hard way” project from Kelsey Hightower. That’ll give you an idea of what it takes and help you understand why IT admins are looking for packaged and easy-to-deploy Kubernetes implementations 😊   But the good news is that SAS provides the SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes project (aka sassoftware/viya4-iac-k8s) to help IT and Kubernetes administrators to create and configure a cluster that meets SAS Viya system requirements. To summarize, what falls under the standard support policy for open-source Kubernetes are clusters that were installed: with the Viya 4 IaC project, or with officially supported installation tools for the upstream K8s project, which are: kubeadm, kops or kubespray AND that follow the documented selective Kubernetes configuration choices (Calico CNI for the network, containerd as the container runtime, Ingress Nginx as the Ingress controller, etc…).   “OK but my customers are using Rancher for their Kubernetes deployments…”   While CNCF-certified and sometimes open source, packaged distributions like Rancher, Mirantis, or VMware Tanzu are very popular, they fall under the SAS “limited support” policy.   The reason is that those vendor packaged offerings usually include value-added features that often conflict with our deployment or at least need to be accounted for by our deployment. While the Kubernetes bits under the covers are the same, the control plane often imposes additional requirements that have an impact on whether or not things will actually work.   If we look at the landscape of Certified Kubernetes, we can see that there were 69 distributions (at the time of this write-up), and the list keeps growing.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Even if a Viya 4deployment could be successful in one of these distributions, such deployment would also fall under the SAS “limited support” policy. SAS will continue to research the possibilities in providing standard official support for other certified Kubernetes distributions. Should my customer use an alternative Kubernetes distributions?   Although not part of the standard supported platforms, various Kubernetes implementations have deployed successfully. SAS Teams have deployed Viya 4 using tools like RKE, Tanzu, ARO, Oracle, OpenShift on IBM Cloud, etc... . This information, as well as running a successful viya4 ARK pre-installation report, increases the level of confidence for customers who want to continue using such distributions.   So yes, for customers wanting the choice of Kubernetes platform such as RKE Rancher, Mirantis or VMWare Tanzu, (assuming it is a CNCF certified platform) they can proceed. The caveat being that they are aware and are in agreement with the SAS Support for Alternative Kubernetes Distributions  policy (which includes the definition of “limited support”). Selective configuration choices   SAS provides standard support for environments that have been internally tested. As it would not be realistic to test every possible combinations of the Kubernetes infrastructure options, there are some key requirements to limit the scope of the possible.   Here is the link to the official documentation.   The new open source Kubernetes support is subject to most of the standard and documented Virtual Infrastructure requirements that are also true for cloud-managed Kubernetes, such as AKS, GKE or EKS.   These requirements includes things like : Kubernetes version, separate subnets for pods and services, dedicated namespace for Viya, Ingress Nginx, node labels, openSSL or cert-manager for TLS certification generation, etc.   But  a few specific requirements for the “upstream” open source Kubernetes are explicitly called out in the documentation: “At this time, Calico 3.23.1 or later is the only software-defined networking layer for Kubernetes that is supported for a SAS Viya deployment in open source Kubernetes.” “To manage the required load balancer IP addresses, a utility like kube-vip or MetalLB is required.” “For a deployment on open source Kubernetes, only a containerd runtime is supported.” “For open source Kubernetes, you must have an A record with the FQDN associated with the load balancer and a CNAME record consisting of a wildcard entry pointing to that A” So let’s review and explain these specific requirements a little bit more.   Calico CNI   As explained on this page, “CNI stands for Container Network Interface, a standard designed to make it easy to configure container networking when containers are created or destroyed. These plugins do the work of making sure that Kubernetes’ networking requirements are satisfied and providing the networking features that cluster administrators require.”   Various CNI  plugins (Flannel, Calico, Weave, Canal, etc ) have been released to address specific environments and requirements. Why do we require the Calico CNI and not something else? Well, it was important to be selective/opiniated on the CNI (to restrict the scope of support) and the two images below shows the performance, security benefits, and popularity of Calico.       Technically Calico allows “inter-node” communications between pods and services inside the cluster by creating a layer-3 network to route packets between hosts and assigning a fully routable IP address to every pod. In addition to networking connectivity, Calico is well-known for its advanced network features. As an example, Calico allows you to define network policies.   When we deploy with the IaC tool for opensource Kubernetes, we can specify distinct CIDR IP ranges for pods and services, such as:   kubernetes_pod_subnet : "10.42.0.0/16" kubernetes_service_subnet : "10.43.0.0/16"   and Calico will create virtual and internal (within the Kubernetes cluster) routable IP addresses from these ranges for the Kubernetes pods and services.   Load-balancer/VIP utility   In a Viya 4 environment, load-balancers with external IP addresses are required to expose Viya services and web applications outside of the cluster. In addition, if we install open source Kubernetes in HA mode with 3 control plane nodes, we also need a load-balancer to route our requests across the 3 nodes.   When Kubernetes runs in the Cloud (Azure, AWS or GCP for example), a control plane component called the “cloud-controller-manager” links your cluster into the cloud provider's API.   So when you create a Kubernetes "LoadBalancer" type of service to expose your applications, the cloud-controller-manager interacts with the Cloud provider to create the load-balancer and external IP resources in the Cloud.     But, if you are running Kubernetes on your own physical or virtual machines (on premises), the cluster does not have a cloud controller manager…   Several solutions exist to solve this issue, such as  kube-vip or MetalLB, which can leverage network protocols (such as ARP or BGP) to simulate similar behavior using a pool of pre-defined floating IP addresses (IP addresses that must be part of the network but are not assigned).   As an example, kube-vip is used by the SAS Viya 4 IaC for Open Source Kubernetes project to use a pre-determined Virtual IP address as the external IP address for the LoadBalancer service.         Container Runtime Interface (CRI)   Another selective/opiniated choice is the Container Runtime (Kubernetes cannot run any container unless it has access to a Container Runtime, such as Docker engine, containerd or CRI-O).   In theory the container runtime does not matter for Kubernetes, but we have seen containerd progressively replace Docker as the standard default container runtime in modern Kubernetes Managed Services (AKS, GCP, etc).   While you can still use Docker to build your standard container images and run them with other container runtimes, using the Docker engine as the container runtime has been deprecated by the Kubernetes project after v1.20, in favor of runtimes that use the Container Runtime Interface (CRI) created for Kubernetes.   So the requirement here is that the customer has installed and configured upstream Open Source Kubernetes with “containerd” as the container runtime. (See this page for details when deploying with kubeadm.)   What else?   Finally, in the official requirements table below:     We can also notice that 4 node pools are required (instead of the 3 node pools that we require for other cloud platforms). The requirement differs because in the Cloud managed Kubernetes services, the control plane is under the Cloud provider’s control, and you don’t have access to it (the Cloud provider is in charge of the maintenance, security, and regular upgrades of the control plane). But in an on-premises implementation of open source Kubernetes, the customers manage the whole cluster infrastructure (including the control plane) – which also means that the they are responsible for maintaining, securing and keeping up to date the Kubernetes environment.   Introducing the new IaC project for open source Kubernetes   The SAS IaC tool for open source Kubernetes allows IT administrators to create a Kubernetes cluster and the associated infrastructure components where customers can then deploy a standard and supported Viya environment.   The diagram below (from the GitHub repository) shows the infrastructure components that can be automatically  provisioned by the tool.       The tool support two types of deployments: bare-metal or vSphere and offers two deployment methods: either bash or docker.   In the bare-metal deployment, the customer must provide their own VMs or physical machines.   In the vSphere deployment, the customer can use terraform to automate the provisioning of the VMware VMs in the vSphere environment.     Then the project’s Ansible playbooks are run to configure the nodes at the system level and to install the Kubernetes Open Source software (using kubeadm behind the scene).   A script ( oss-k8s.sh ) calls the Ansible playbooks to perform these tasks.   The diagram, below summarizes the process.         So, as with other IaC projects, you can run the tool in standard mode (bash) or inside a Docker container.   Using the Docker container requires you to build the Docker image of the tool, and then call it to perform the Terraform provisioning (for a vSphere deployment), node configuration, and K8s installation.   The benefits of using the docker container method is that you don’t have to install all the required third-party tools (such as Ansible, Helm, Terraform, or kubectl) because they are included inside the Docker image.       Conclusion   In this first part of the blog, we discussed the scope of support, opiniated requirements and provided a high-level description of the Viya IaC tool for open source Kubernetes.   In the next part of the blog, we will look into the specific requirements for the IaC tool, see how to configure it for a "bare metal" type deployment (with the Ansible inventory and variables) and share some first "hands-on" experience feedback from our utilization of the tool.   Finally, we’ll show how it can be combined with the Deployment as Code project to prepare the "baseline" and deploy Viya in our open source Kubernetes cluster in an automated way.   Find more articles from SAS Global Enablement and Learning here. ... View more