Kubernetes Deployment Issues - LTS 2023.10

[Narendra_P]

Executed the Kubernetes deployment for all components, and while some pods are successfully running, we are facing issues with others:

1. Some pods are in a waiting state, awaiting the readiness of the Consul server, but we are encountering the following error.

FYI : I have gone through the below link for the above error. but i am unable to find the file or path to fix it in LTS 2O23.10 release.

SAS VIYA 3.4 : https://communities.sas.com/t5/SAS-Viya/SAS-Viya-Installation/td-p/495208

   2. Other pods are awaiting for SAS folders.

Could you please suggest on these issues?

[gwootton]

I think this will occur if your cluster's configured internal network isn't set up to use a range within the private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).

[Narendra_P]

Thank you for your prompt response.

Are both issue 1 and issue 2 related to the fact that the configured internal network for the cluster doesn't fall within the recommended private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)? If so, is there an alternative solution available? Additionally, the documentation doesn't seem to provide guidance on using these specified IP ranges.

[gwootton]

Yes, most services are dependent on Consul, so if consul is not able to start they will not be able to enter a ready state.

I'm not aware of any way to circumvent this requirement.

I found a possible solution would be to set the environment variable "CONSUL_BIND_EXTERNAL" to "eth0".

Does running this command allow the consul server to start?

kubectl -n namespace set env sts/sas-consul-server CONSUL_BIND_EXTERNAL=eth0

If so, this would also need to be done in the CAS configuration, so you could use these two patchTransformers in your deployment, then rebuild/apply:

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-consul-bind-transformer
patch: |-
  - op: add
    path: /spec/template/spec/containers/0/env/-
    value:
      name: CONSUL_BIND_EXTERNAL
      value: eth0
target:
  group: apps
  kind: StatefulSet
  name: sas-consul-server
  version: v1
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-cas-consul-bind-transformer
patch: |-
  - op: add
    path: /spec/controllerTemplate/spec/containers/2/env/-
    value:
      name: CONSUL_BIND_EXTERNAL
      value: eth0
target:
  group: viya.sas.com
  kind: CASDeployment
  name: .*
  version: v1alpha1

[Narendra_P]

Thank you @gwootton ! Your assistance and advice are greatly valued.

Following the execution of the command below, the sas-consul server has been initiated.

kubectl -n namespace set env sts/sas-consul-server CONSUL_BIND_EXTERNAL=eth0

And Included the two patchTransformers in the deployment under the site-config directory, following the structure mentioned below, and executed a rebuild.

└── $deploy/
    ├── kustomization.yaml
    ├── sas-bases/
    └── site-config/

The issue with SAS folders persists even after these steps. Your insights on this matter would be greatly appreciated.

Here is the current status of the pods, primarily in a waiting state for SAS folders.

[Narendra_P]

++

All of the pods are currently in the initialization state, and the sas-start-sequencer container has finished but is in a pending state within the sas-certframe container. The logs for the pod indicate that it is waiting for SAS folders.

For instance,

[gwootton]

The sas-start-sequencer initcontainer orchestrates startup of services such that services don't start until the services they depend on have been started. sas-certframe typically does not depend on other Viya services, it's job is to create the Viya TLS truststore and generate the certificates used to secure communication between the pods.
If most services indicate they are waiting on SAS Folders, you may wish to check what SAS Folders is waiting on. sas-logon-app and sas-configuration are relied on by most all pods.

[Narendra_P]

Thank You! @gwootton 

"sas-folders" is currently awaiting the availability of "sas-logon."

The deployment status of both "sas-configuration" and "sas-logon"  pods are in a pending state, and this delay is attributed to the "DB check sleeping" condition (see below). In pod logs, apart from the "DB check sleeping", not able to find the sufficient information to address the problem.

Could you please suggest on this?

FYI:

-> Validated the postgres dataserver details.

└── $deploy/
    ├── kustomization.yaml
    ├── sas-bases/
    └── site-config/postgres

-> Performed connectivity test from GKE node to SQL database. (see below)

[gwootton]

This suggests the database is not accessible by the sas-logon and sas-configuration pods using the provided connection information.

It sounds like you are using external postgres and maybe have not performed all the required steps for configuring external postgres. This is described in:

$deploy/sas-bases/examples/postgres/README.md under "External PostgreSQL Configuration". There are additional specific steps for Google Cloud Platform Cloud SQL for PostgreSQL if this is what you are using.

[Narendra_P]

Hi @gwootton ,

I suspect there could be an issue with the connection to the database through the "cloud_sql_proxy." Upon inspecting the platform-postgres-sql-proxy pod, I noticed that despite the pod being active, but the logs indicate the following:

[gwootton]

Looks like it's failing to pull the configuration file from googleapis.com, so I'd guess this is related to the provided ServiceAccountKey. You may want to engage Technical Support.

[Narendra_P]

Hi @gwootton ,

Could you please suggest on below issues:

Note: Using Internal postgres.

1) sas-model-repository:

2)sas-search:

3)sas-cas-server-default-controller:

Note:

I've configured the environment variable "CONSUL_BIND_EXTERNAL" to "eth0" in an attempt to address the "No Private IPv4 address found" error. However, the issue persists.

[Narendra_P]

Also, could you please inform me about the purpose or functionality of the following components?
sas-search,
sas-model-repository and
sas-cas-server-default-controller
sas-cas-control

[gwootton]

sas-search is the microservice that facilitates the search function. sas-model-repository is used to manage models. sas-cas-server-default-controller is the CAS controller. sas-cas-control is the microservice that enables management of the CAS server through a REST API.

[gwootton · Accepted solution]

It looks like sas-search and sas-model-repository are unable to contact CAS to bootstrap their CASLIBs, and CAS is failing to start because the cosul sidecar isn't able to start. After using the patch I provided for the CASDeployment object, did you delete the cas-server-default-controller pod so it gets recreated with the new definition?