I am currently experiencing an issue with deploying our application due to difficulties in pulling images from the SAS website. It appears that the source of the files is down, which is causing the deployment process to halt.
[root@masterocp ~]# oc get pods -n sas-viya NAME READY STATUS RESTARTS AGE sas-deployment-operator-reconcile 0/1 Completed 0 159m sas-deployment-operator-reconcile-3787d8f7-sc8lm 1/1 Running 0 11m sas-prepull-79b4b65749-9qq6v 0/1 ImagePullBackOff 0 6m45s
[root@masterocp ~]# oc describe pod sas-prepull-79b4b65749-9qq6v Name: sas-prepull-79b4b65749-9qq6v Namespace: sas-viya Priority: 0 Node: nodo1/172.**.**.** Start Time: Tue, 18 Jun 2024 14:34:59 -0400 Labels: app=sas-prepull app.kubernetes.io/name=sas-prepull pod-template-hash=79b4b65749 sas.com/deployment=sas-viya workload.sas.com/class=stateless Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.1.84" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.1.84" ], "default": true, "dns": {} }] openshift.io/scc: restricted-v2 prometheus.io/scrape: false sas.com/certificate-file-format: none sas.com/component-name: sas-prepull sas.com/component-version: 2.5.23-20240108.1704720802223 sas.com/kustomize-base: base sas.com/version: 2.5.23 seccomp.security.alpha.kubernetes.io/pod: runtime/default sidecar.istio.io/inject: false Status: Pending IP: 10.130.1.84 IPs: IP: 10.130.1.84 Controlled By: ReplicaSet/sas-prepull-79b4b65749 Containers: sas-prepull: Container ID: Image: cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223 Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Limits: cpu: 1 memory: 1Gi Requests: cpu: 50m memory: 50Mi Environment: SAS_PREPULL_NAMESPACE: sas-viya (v1:metadata.namespace) SAS_SPEC_NODENAME: (v1:spec.nodeName) Mounts: /tmp from tmp (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58n2f (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-58n2f: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s workload.sas.com/class=stateful:NoSchedule workload.sas.com/class=stateless:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 7m default-scheduler Successfully assigned sas-viya/sas-prepull-79b4b65749-9qq6v to nodo1 Normal AddedInterface 6m59s multus Add eth0 [10.130.1.84/22] from openshift-sdn Normal Pulling 5m24s (x4 over 6m59s) kubelet Pulling image "cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223" Warning Failed 5m23s (x4 over 6m56s) kubelet Failed to pull image "cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223": rpc error: code = Unknown desc = reading manifest 2.5.23-20240108.1704720802223 in cr.sas.com/sas-prepull: StatusCode: 404, 404 page not found Warning Failed 5m23s (x4 over 6m56s) kubelet Error: ErrImagePull Warning Failed 5m9s (x6 over 6m55s) kubelet Error: ImagePullBackOff Normal BackOff 115s (x19 over 6m55s) kubelet Back-off pulling image "cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223"
Can you help me with this case?
thanks in advance
Hi @etrock
This would usually mean the image cannot be pulled from your source registry "cr.sas.com"
cr.sas.com/sas-prepull:2.5.23-20240108.1704720802223
Thanks for you replied @AnandVyas I know that is the issue . What I can't understand is that the network has full access to the Internet without restrictions of firewalls.
I have no experience pulling directly from cr.sas.com apart from one Docker image or two using docker cli. Unless your datacenter hosting OCP cluster is located near cr.sas.com, this is not a recommended practice and you should use a mirror registry instead. Viya docker images for a deployment might represent 30-50Gb globally, and some images are individually huge (3-8 Gb). Therefore, performance would suffer each time the kubelet needs refreshing the image, this would add extra even noticeable delays for your endusers. Set up a registry and populate it using mirror manager. There are even mirror registries provided with OCP.
I suspect you encounter some binding/authentication issue with cr.sas.com. Try pulling the SAS OQ Docker image : (https://documentation.sas.com/doc/en/sasadmincdc/v_053/oq/n1preqegzpz85rn1c7a6j96ewr9f.htm)
If you succeed, then try pulling one image or two from your deployment. If it works then you might try creating a manual image pull secret in your Viya Namespace : see https://docs.openshift.com/container-platform/4.14/openshift_images/managing_images/using-image-pull... and configure it with the login/pwd used previously.
The SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment.
SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.
Find more tutorials on the SAS Users YouTube channel.