Viya 4 Pods does not start after restart

[alko13]

Hi

We have restarted our sas viya environment, however after restart alot of pods are not running/completed.

Some pods contain this type of messages:

This one is from sas-crunchy-data-postgres-operator describe output

  Normal   Pulling         11m (x4 over 16m)   kubelet            Pulling image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.25.5-20220408.1649425759549"

  Warning  Failed          10m (x4 over 15m)   kubelet            Failed to pull image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.25.5-20220408.1649425759549": rpc error: code = Unknown desc = pinging container registry cr.sas.com: Get "https://cr.sas.com/v2/": dial tcp 149.173.160.155:443: i/o timeout

  Warning  Failed          10m (x4 over 15m)   kubelet            Error: ErrImagePull

  Warning  Failed          10m (x5 over 15m)   kubelet            Error: ImagePullBackOff

  Normal   BackOff         72s (x38 over 15m)  kubelet            Back-off pulling image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.25.5-20220408.1649425759549"

I tried to telnet cr.sas.com 443 from the bastion host and seems fine.

Attaching as well the describe pods, get pods output

Appreciate  any advice,

Thanks!

[JosvanderVelden]

In cases like this it is always a good idea to contact SAS technical asap. https://support.sas.com/en/technical-support/contact-sas.lang.html

Provide as much detail as you can about the environment/setup/configuration and complete zipped logfiles. They will ask for them anyway.

[gwootton · Accepted solution]

This message is saying kubernetes is not receiving a response when trying to connect to cr.sas.com to download images. That you can connect from your bastion host proves the issue is not with cr.sas.com but with your infrastructure. I suspect the firewall configuration needs to be updated to allow these outbound connections from the nodes.