topic Re: Viya 4 Pods does not start after restart in Administration and Deployment

Viya 4 Pods does not start after restart

alko13 — Tue, 07 Mar 2023 09:32:20 GMT

We have restarted our sas viya environment, however after restart alot of pods are not running/completed.

Some pods contain this type of messages:

This one is from sas-crunchy-data-postgres-operator describe output

Normal Pulling 11m (x4 over 16m) kubelet Pulling image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.25.5-20220408.1649425759549"

Warning Failed 10m (x4 over 15m) kubelet Failed to pull image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.25.5-20220408.1649425759549": rpc error: code = Unknown desc = pinging container registry cr.sas.com: Get "https://cr.sas.com/v2/": dial tcp 149.173.160.155:443: i/o timeout

Warning Failed 10m (x4 over 15m) kubelet Error: ErrImagePull

Warning Failed 10m (x5 over 15m) kubelet Error: ImagePullBackOff

Normal BackOff 72s (x38 over 15m) kubelet Back-off pulling image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.25.5-20220408.1649425759549"

I tried to telnet cr.sas.com 443 from the bastion host and seems fine.

Attaching as well the describe pods, get pods output

Appreciate any advice,

Thanks!

Re: Viya 4 Pods does not start after restart

JosvanderVelden — Tue, 07 Mar 2023 11:03:13 GMT

In cases like this it is always a good idea to contact SAS technical asap. https://support.sas.com/en/technical-support/contact-sas.lang.html

Provide as much detail as you can about the environment/setup/configuration and complete zipped logfiles. They will ask for them anyway.

Re: Viya 4 Pods does not start after restart

gwootton — Tue, 07 Mar 2023 14:43:15 GMT

This message is saying kubernetes is not receiving a response when trying to connect to cr.sas.com to download images. That you can connect from your bastion host proves the issue is not with cr.sas.com but with your infrastructure. I suspect the firewall configuration needs to be updated to allow these outbound connections from the nodes.