ImagePullBackOff Error on Deploying SAS VIYA 4 in OCP

[etrock]

I am currently experiencing an issue with deploying our application due to difficulties in pulling images from the SAS website. It appears that the source of the files is down, which is causing the deployment process to halt. 

[root@masterocp ~]# oc get pods -n sas-viya
NAME                                               READY   STATUS             RESTARTS   AGE
sas-deployment-operator-reconcile                  0/1     Completed          0          159m
sas-deployment-operator-reconcile-3787d8f7-sc8lm   1/1     Running            0          11m
sas-prepull-79b4b65749-9qq6v                       0/1     ImagePullBackOff   0          6m45s

[root@masterocp ~]# oc describe pod sas-prepull-79b4b65749-9qq6v
Name:         sas-prepull-79b4b65749-9qq6v
Namespace:    sas-viya
Priority:     0
Node:         nodo1/172.**.**.**
Start Time:   Tue, 18 Jun 2024 14:34:59 -0400
Labels:       app=sas-prepull
              app.kubernetes.io/name=sas-prepull
              pod-template-hash=79b4b65749
              sas.com/deployment=sas-viya
              workload.sas.com/class=stateless
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.130.1.84"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.130.1.84"
                    ],
                    "default": true,
                    "dns": {}
                }]
              openshift.io/scc: restricted-v2
              prometheus.io/scrape: false
              sas.com/certificate-file-format: none
              sas.com/component-name: sas-prepull
              sas.com/component-version: 2.5.23-20240108.1704720802223
              sas.com/kustomize-base: base
              sas.com/version: 2.5.23
              seccomp.security.alpha.kubernetes.io/pod: runtime/default
              sidecar.istio.io/inject: false
Status:       Pending
IP:           10.130.1.84
IPs:
  IP:           10.130.1.84
Controlled By:  ReplicaSet/sas-prepull-79b4b65749
Containers:
  sas-prepull:
    Container ID:   
    Image:          cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     50m
      memory:  50Mi
    Environment:
      SAS_PREPULL_NAMESPACE:  sas-viya (v1:metadata.namespace)
      SAS_SPEC_NODENAME:       (v1:spec.nodeName)
    Mounts:
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58n2f (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-58n2f:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                             workload.sas.com/class=stateful:NoSchedule
                             workload.sas.com/class=stateless:NoSchedule
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       7m                     default-scheduler  Successfully assigned sas-viya/sas-prepull-79b4b65749-9qq6v to nodo1
  Normal   AddedInterface  6m59s                  multus             Add eth0 [10.130.1.84/22] from openshift-sdn
  Normal   Pulling         5m24s (x4 over 6m59s)  kubelet            Pulling image "cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223"
  Warning  Failed          5m23s (x4 over 6m56s)  kubelet            Failed to pull image "cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223": rpc error: code = Unknown desc = reading manifest 2.5.23-20240108.1704720802223 in cr.sas.com/sas-prepull: StatusCode: 404, 404 page not found
  Warning  Failed          5m23s (x4 over 6m56s)  kubelet            Error: ErrImagePull
  Warning  Failed          5m9s (x6 over 6m55s)   kubelet            Error: ImagePullBackOff
  Normal   BackOff         115s (x19 over 6m55s)  kubelet            Back-off pulling image "cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223"

Can you help me with this case?

thanks in advance

[Anand_V]

Hi @etrock 

This would usually mean the image cannot be pulled from your source registry "cr.sas.com"

cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223

[etrock]

Thanks for you replied @Anand_V I know that is the issue . What I can't understand is that the network has full access to the Internet without restrictions of firewalls.

[Ronan_Lincoln]

I have no experience pulling directly from cr.sas.com apart from one Docker image or two using docker cli. Unless your datacenter hosting OCP cluster is located near cr.sas.com, this is not a recommended practice and you should use a mirror registry instead. Viya docker images for a deployment might represent 30-50Gb globally, and some images are individually huge (3-8 Gb). Therefore, performance would suffer each time the kubelet needs refreshing the image, this would add extra even noticeable delays for your endusers. Set up a registry and populate it using mirror manager. There are even mirror registries provided with OCP.

I suspect you encounter some binding/authentication issue with cr.sas.com. Try pulling the SAS OQ Docker image : (https://documentation.sas.com/doc/en/sasadmincdc/v_053/oq/n1preqegzpz85rn1c7a6j96ewr9f.htm)

If you succeed, then try pulling one image or two from your deployment. If it works then you might try creating a manual image pull secret in your Viya Namespace : see https://docs.openshift.com/container-platform/4.14/openshift_images/managing_images/using-image-pull... and configure it with the login/pwd used previously.