BookmarkSubscribeRSS Feed
☑ This topic is solved. Need further help from the community? Please sign in and ask a new question.
Narendra_P
Fluorite | Level 6

Executed the Kubernetes deployment for all components, and while some pods are successfully running, we are facing issues with others:

 

  1. Some pods are in a waiting state, awaiting the readiness of the Consul server, but we are encountering the following error.

 

Narendra_P_0-1701355647514.png

 

FYI : I have gone through the below link for the above error. but i am unable to find the file or path to fix it in LTS 2O23.10 release.

SAS VIYA 3.4 : https://communities.sas.com/t5/SAS-Viya/SAS-Viya-Installation/td-p/495208

 

   2. Other pods are awaiting for SAS folders.

 

Narendra_P_1-1701355647540.png

 

 

Could you please suggest on these issues?

1 ACCEPTED SOLUTION

Accepted Solutions
gwootton
SAS Super FREQ
It looks like sas-search and sas-model-repository are unable to contact CAS to bootstrap their CASLIBs, and CAS is failing to start because the cosul sidecar isn't able to start. After using the patch I provided for the CASDeployment object, did you delete the cas-server-default-controller pod so it gets recreated with the new definition?
--
Greg Wootton | Principal Systems Technical Support Engineer

View solution in original post

16 REPLIES 16
gwootton
SAS Super FREQ
I think this will occur if your cluster's configured internal network isn't set up to use a range within the private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).
--
Greg Wootton | Principal Systems Technical Support Engineer
Narendra_P
Fluorite | Level 6

Thank you for your prompt response.


Are both issue 1 and issue 2 related to the fact that the configured internal network for the cluster doesn't fall within the recommended private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)? If so, is there an alternative solution available? Additionally, the documentation doesn't seem to provide guidance on using these specified IP ranges.

gwootton
SAS Super FREQ

Yes, most services are dependent on Consul, so if consul is not able to start they will not be able to enter a ready state.

I'm not aware of any way to circumvent this requirement.

 

I found a possible solution would be to set the environment variable "CONSUL_BIND_EXTERNAL" to "eth0".

 

Does running this command allow the consul server to start?

kubectl -n namespace set env sts/sas-consul-server CONSUL_BIND_EXTERNAL=eth0

 

If so, this would also need to be done in the CAS configuration, so you could use these two patchTransformers in your deployment, then rebuild/apply:

 

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-consul-bind-transformer
patch: |-
  - op: add
    path: /spec/template/spec/containers/0/env/-
    value:
      name: CONSUL_BIND_EXTERNAL
      value: eth0
target:
  group: apps
  kind: StatefulSet
  name: sas-consul-server
  version: v1
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-cas-consul-bind-transformer
patch: |-
  - op: add
    path: /spec/controllerTemplate/spec/containers/2/env/-
    value:
      name: CONSUL_BIND_EXTERNAL
      value: eth0
target:
  group: viya.sas.com
  kind: CASDeployment
  name: .*
  version: v1alpha1

 

--
Greg Wootton | Principal Systems Technical Support Engineer
Narendra_P
Fluorite | Level 6

Thank you @gwootton ! Your assistance and advice are greatly valued.

 

Following the execution of the command below, the sas-consul server has been initiated.

 

kubectl -n namespace set env sts/sas-consul-server CONSUL_BIND_EXTERNAL=eth0

Narendra_P_2-1701383534266.png

 

And Included the two patchTransformers in the deployment under the site-config directory, following the structure mentioned below, and executed a rebuild.

└── $deploy/
    ├── kustomization.yaml
    ├── sas-bases/
    └── site-config/

 

The issue with SAS folders persists even after these steps. Your insights on this matter would be greatly appreciated.

 

Narendra_P_0-1701383378606.png


Here is the current status of the pods, primarily in a waiting state for SAS folders.

Narendra_P_3-1701385807046.png

Narendra_P_4-1701385863338.png

Narendra_P_5-1701386002190.png

Narendra_P_6-1701386045702.png

 

 

Narendra_P
Fluorite | Level 6

++

All of the pods are currently in the initialization state, and the sas-start-sequencer container has finished but is in a pending state within the sas-certframe container. The logs for the pod indicate that it is waiting for SAS folders.

 

For instance,

 

Narendra_P_0-1701439323137.png

 

Narendra_P_1-1701439727967.png

Narendra_P_2-1701441891242.png

 

gwootton
SAS Super FREQ
The sas-start-sequencer initcontainer orchestrates startup of services such that services don't start until the services they depend on have been started. sas-certframe typically does not depend on other Viya services, it's job is to create the Viya TLS truststore and generate the certificates used to secure communication between the pods.
If most services indicate they are waiting on SAS Folders, you may wish to check what SAS Folders is waiting on. sas-logon-app and sas-configuration are relied on by most all pods.
--
Greg Wootton | Principal Systems Technical Support Engineer
Narendra_P
Fluorite | Level 6

Thank You! @gwootton 

 

"sas-folders" is currently awaiting the availability of "sas-logon."

Narendra_P_1-1701676171165.png

 

The deployment status of both "sas-configuration" and "sas-logon"  pods are in a pending state, and this delay is attributed to the "DB check sleeping" condition (see below). In pod logs, apart from the "DB check sleeping", not able to find the sufficient information to address the problem.

 

Narendra_P_0-1701675958785.png

 

Could you please suggest on this?

 

FYI:

-> Validated the postgres dataserver details.

└── $deploy/
    ├── kustomization.yaml
    ├── sas-bases/
    └── site-config/postgres

-> Performed connectivity test from GKE node to SQL database. (see below)

Narendra_P_3-1701685209985.png

 

 

gwootton
SAS Super FREQ

This suggests the database is not accessible by the sas-logon and sas-configuration pods using the provided connection information.

It sounds like you are using external postgres and maybe have not performed all the required steps for configuring external postgres. This is described in:

$deploy/sas-bases/examples/postgres/README.md under "External PostgreSQL Configuration". There are additional specific steps for Google Cloud Platform Cloud SQL for PostgreSQL if this is what you are using.

--
Greg Wootton | Principal Systems Technical Support Engineer
Narendra_P
Fluorite | Level 6

Hi @gwootton ,

 

I suspect there could be an issue with the connection to the database through the "cloud_sql_proxy." Upon inspecting the platform-postgres-sql-proxy pod, I noticed that despite the pod being active, but the logs indicate the following:

 

Narendra_P_0-1701864428272.png

 

gwootton
SAS Super FREQ
Looks like it's failing to pull the configuration file from googleapis.com, so I'd guess this is related to the provided ServiceAccountKey. You may want to engage Technical Support.
--
Greg Wootton | Principal Systems Technical Support Engineer
Narendra_P
Fluorite | Level 6

Hi @gwootton ,

 

Could you please suggest on below issues:

Note: Using Internal postgres.

 

1) sas-model-repository:

Narendra_P_3-1701990250739.png

Narendra_P_1-1701990070026.png

Narendra_P_2-1701990172125.png

 

2)sas-search:

Narendra_P_4-1701990906580.png

Narendra_P_5-1701990933682.png

 

3)sas-cas-server-default-controller:

Narendra_P_6-1701992668704.png

Note:

I've configured the environment variable "CONSUL_BIND_EXTERNAL" to "eth0" in an attempt to address the "No Private IPv4 address found" error. However, the issue persists.

Narendra_P
Fluorite | Level 6
Also, could you please inform me about the purpose or functionality of the following components?
sas-search,
sas-model-repository and
sas-cas-server-default-controller
sas-cas-control
gwootton
SAS Super FREQ
sas-search is the microservice that facilitates the search function. sas-model-repository is used to manage models. sas-cas-server-default-controller is the CAS controller. sas-cas-control is the microservice that enables management of the CAS server through a REST API.
--
Greg Wootton | Principal Systems Technical Support Engineer
gwootton
SAS Super FREQ
It looks like sas-search and sas-model-repository are unable to contact CAS to bootstrap their CASLIBs, and CAS is failing to start because the cosul sidecar isn't able to start. After using the patch I provided for the CASDeployment object, did you delete the cas-server-default-controller pod so it gets recreated with the new definition?
--
Greg Wootton | Principal Systems Technical Support Engineer

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

CLI in SAS Viya

Learn how to install the SAS Viya CLI and a few commands you may find useful in this video by SAS’ Darrell Barton.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 16 replies
  • 2126 views
  • 2 likes
  • 2 in conversation