topic Re: Kubernetes Deployment Issues - LTS 2023.10 in Administration and Deployment

Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Thu, 30 Nov 2023 14:51:25 GMT

Executed the Kubernetes deployment for all components, and while some pods are successfully running, we are facing issues with others:

Some pods are in a waiting state, awaiting the readiness of the Consul server, but we are encountering the following error.

FYI : I have gone through the below link for the above error. but i am unable to find the file or path to fix it in LTS 2O23.10 release.

SAS VIYA 3.4 : https://communities.sas.com/t5/SAS-Viya/SAS-Viya-Installation/td-p/495208

2. Other pods are awaiting for SAS folders.

Could you please suggest on these issues?

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Thu, 30 Nov 2023 17:44:33 GMT

I think this will occur if your cluster's configured internal network isn't set up to use a range within the private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Thu, 30 Nov 2023 18:00:17 GMT

Thank you for your prompt response.

Are both issue 1 and issue 2 related to the fact that the configured internal network for the cluster doesn't fall within the recommended private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)? If so, is there an alternative solution available? Additionally, the documentation doesn't seem to provide guidance on using these specified IP ranges.

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Thu, 30 Nov 2023 19:45:01 GMT

Yes, most services are dependent on Consul, so if consul is not able to start they will not be able to enter a ready state.

~~I'm not aware of any way to circumvent this requirement.~~

I found a possible solution would be to set the environment variable "CONSUL_BIND_EXTERNAL" to "eth0".

Does running this command allow the consul server to start?

kubectl -n namespace set env sts/sas-consul-server CONSUL_BIND_EXTERNAL=eth0

If so, this would also need to be done in the CAS configuration, so you could use these two patchTransformers in your deployment, then rebuild/apply:

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-consul-bind-transformer
patch: |-
  - op: add
    path: /spec/template/spec/containers/0/env/-
    value:
      name: CONSUL_BIND_EXTERNAL
      value: eth0
target:
  group: apps
  kind: StatefulSet
  name: sas-consul-server
  version: v1
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-cas-consul-bind-transformer
patch: |-
  - op: add
    path: /spec/controllerTemplate/spec/containers/2/env/-
    value:
      name: CONSUL_BIND_EXTERNAL
      value: eth0
target:
  group: viya.sas.com
  kind: CASDeployment
  name: .*
  version: v1alpha1

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Thu, 30 Nov 2023 23:17:06 GMT

Thank you @gwootton ! Your assistance and advice are greatly valued.

Following the execution of the command below, the sas-consul server has been initiated.

kubectl -n namespace set env sts/sas-consul-server CONSUL_BIND_EXTERNAL=eth0

And Included the two patchTransformers in the deployment under the site-config directory, following the structure mentioned below, and executed a rebuild.

└── $deploy/
    ├── kustomization.yaml
    ├── sas-bases/
    └── site-config/

The issue with SAS folders persists even after these steps. Your insights on this matter would be greatly appreciated.

Here is the current status of the pods, primarily in a waiting state for SAS folders.

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Fri, 01 Dec 2023 14:45:39 GMT

All of the pods are currently in the initialization state, and the sas-start-sequencer container has finished but is in a pending state within the sas-certframe container. The logs for the pod indicate that it is waiting for SAS folders.

For instance,

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Fri, 01 Dec 2023 14:51:56 GMT

The sas-start-sequencer initcontainer orchestrates startup of services such that services don't start until the services they depend on have been started. sas-certframe typically does not depend on other Viya services, it's job is to create the Viya TLS truststore and generate the certificates used to secure communication between the pods.
If most services indicate they are waiting on SAS Folders, you may wish to check what SAS Folders is waiting on. sas-logon-app and sas-configuration are relied on by most all pods.

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Mon, 04 Dec 2023 10:23:26 GMT

Thank You! @gwootton

"sas-folders" is currently awaiting the availability of "sas-logon."

The deployment status of both "sas-configuration" and "sas-logon" pods are in a pending state, and this delay is attributed to the "DB check sleeping" condition (see below). In pod logs, apart from the "DB check sleeping", not able to find the sufficient information to address the problem.

Could you please suggest on this?

FYI:

-> Validated the postgres dataserver details.

└── $deploy/
    ├── kustomization.yaml
    ├── sas-bases/
    └── site-config/postgres

-> Performed connectivity test from GKE node to SQL database. (see below)

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Mon, 04 Dec 2023 14:00:22 GMT

This suggests the database is not accessible by the sas-logon and sas-configuration pods using the provided connection information.

It sounds like you are using external postgres and maybe have not performed all the required steps for configuring external postgres. This is described in:

$deploy/sas-bases/examples/postgres/README.md under "External PostgreSQL Configuration". There are additional specific steps for Google Cloud Platform Cloud SQL for PostgreSQL if this is what you are using.

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Wed, 06 Dec 2023 12:47:02 GMT

Hi @gwootton ,

I suspect there could be an issue with the connection to the database through the "cloud_sql_proxy." Upon inspecting the platform-postgres-sql-proxy pod, I noticed that despite the pod being active, but the logs indicate the following:

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Wed, 06 Dec 2023 14:40:23 GMT

Looks like it's failing to pull the configuration file from googleapis.com, so I'd guess this is related to the provided ServiceAccountKey. You may want to engage Technical Support.

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Thu, 07 Dec 2023 23:52:07 GMT

Hi @gwootton ,

Could you please suggest on below issues:

Note: Using Internal postgres.

1) sas-model-repository:

2)sas-search:

3)sas-cas-server-default-controller:

Note:

I've configured the environment variable "CONSUL_BIND_EXTERNAL" to "eth0" in an attempt to address the "No Private IPv4 address found" error. However, the issue persists.

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Fri, 08 Dec 2023 12:14:45 GMT

Also, could you please inform me about the purpose or functionality of the following components?
sas-search,
sas-model-repository and
sas-cas-server-default-controller
sas-cas-control

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Fri, 08 Dec 2023 13:42:46 GMT

It looks like sas-search and sas-model-repository are unable to contact CAS to bootstrap their CASLIBs, and CAS is failing to start because the cosul sidecar isn't able to start. After using the patch I provided for the CASDeployment object, did you delete the cas-server-default-controller pod so it gets recreated with the new definition?

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Fri, 08 Dec 2023 13:46:15 GMT

sas-search is the microservice that facilitates the search function. sas-model-repository is used to manage models. sas-cas-server-default-controller is the CAS controller. sas-cas-control is the microservice that enables management of the CAS server through a REST API.

Re: Kubernetes Deployment Issues - LTS 2023.10

Narendra_P — Mon, 11 Dec 2023 20:06:18 GMT

Hi @gwootton ,

Thank you for all the help!

While attempting to access SAS Studio, I encountered the following error:

"A connection to the SAS Compute Server was unavailable."

I have checked the logs of the relevant components. see below:

sas-compute:
A session has been requested for the context with the ID "9c1c50f9-33e1-43a2-b615-7ebad9c0d0b1".

sas-launcher:
caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":10580,"message":"The identifier values for USER \"sasboot\" were not found.","details":["traceId: 9e62dc6d6980982c","path: /identities/users/@currentUser/identifier"],"links":[],"version":2,"httpStatusCode":404}"

sas-studio:
The compute session could not be created.

The documentation specifies that the SAS Viya platform supports only LDAP, Kerberos, and single sign-on methods for authenticating users who are signing in to the environment. Is it not possible to establish user accounts with passwords instead?

Could you please suggest on this?

Re: Kubernetes Deployment Issues - LTS 2023.10

gwootton — Mon, 11 Dec 2023 20:15:59 GMT

This error is because sasboot is a special account for Viya configuration, it cannot be used with SAS Studio.
You would need to log in to Viya with a normal user account.
You cannot add users to Viya directly, only through identity provisioning using LDAP or SCIM. You can authenticate to Viya using LDAP (providing a user ID and password), Kerberos, or SSO like OIDC and SAML.