BookmarkSubscribeRSS Feed
drahorg
Obsidian | Level 7

In SAS Viya 4 (2024.09LTS) I captured in the Opensearch "kubernetes Events" dashboards a few warnings that point to the readiness to sas-formats pod (see below).

During this time our users report a couple of minutes where they cannot connect to SAS Studio.

However, "kubectl describe" or "kubectl logs" for the sas-formats pod does not show anything.

Can you pls help how to find the root cause for this, and potentially fix it.

the json of the event from Opensearch is below

"

{
"_index": "viya_logs-viya-prod-eks-ns-2025-08-20",
"_id": "1030066a-9ad1-429f-b8a3-1f52d7363401",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2025-08-20T11:54:58.000Z",
"level": "Warning",
"message": "Readiness probe failed: Get \"http://100.64.14.80:9080/formats/live\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)",
"logsource": "KUBE_EVENT",
"properties": {
"reason": "Unhealthy",
"eventid": "1030066a-9ad1-429f-b8a3-1f52d7363401",
"firsttimestamp": "2025-08-02T05:33:18Z",
"reportingInstance": "ip-10-221-14-200.ca-centra-1.compute.internal",
"object_type": "Pod",
"lastimestamp": "2025-08-20T11:54:58Z",
"count": 5,
"fieldPath": "spec.containers{sas-portable-formats}",
"eventName": "sas-formats-66f6bb44f9-mmnpd.1857dc4bfe06b0b2",
"type": "Warning",
"reportingComponent": "kubelet",
"object": "sas-formats-66f6bb44f9-mmnpd"
},
"kube": {
"pod": "sas-formats-66f6bb44f9-mmnpd",
"namespace": "viya-prod-eks-ns",
"host": "ip-10-220-13-168.ca-central-1.compute.internal",
"pod_id": "1cd18fa0-ec58-4a3c-af6a-4f0f4d6c2581"
}
},
"fields": {
"@timestamp": [
"2025-08-20T11:54:58.000Z"
]
},
"highlight": {
"logsource.keyword": [
"@opensearch-dashboards-highlighted-field@KUBE_EVENT@/opensearch-dashboards-highlighted-field@"
]
},
"sort": [
1755690898000
]
}

"

Thanks a lot,

3 REPLIES 3
gwootton
SAS Super FREQ
If the liveness probe fails enough, Kubernetes will restart the pod, so you may need to use kubectl logs --previous to see the logs that include the failure.
--
Greg Wootton | Principal Systems Technical Support Engineer
drahorg
Obsidian | Level 7

Thank you Greg for your reply. the SAS Formats pod does not restart. Also both kubectl describe pod and kubectl logs pod does not show any error. I capture this in Opensearch messages showing as error. 

gwootton
SAS Super FREQ
If the pod did not restart then the liveness probe didn't fail enough to trigger this, suggesting the failure was not ongoing. For sas formats the liveness probe by default is queried every 20 seconds, and if it fails three times the service is restarted.
So it sounds like whatever the issue was, it did not last more than 60 seconds.
--
Greg Wootton | Principal Systems Technical Support Engineer

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 491 views
  • 0 likes
  • 2 in conversation