topic Re: SAS formats shows warning and kubernetes and determine couple minutes slowness in Administration and Deployment

SAS formats shows warning and kubernetes and determine couple minutes slowness

drahorg — Thu, 21 Aug 2025 16:21:12 GMT

In SAS Viya 4 (2024.09LTS) I captured in the Opensearch "kubernetes Events" dashboards a few warnings that point to the readiness to sas-formats pod (see below).

During this time our users report a couple of minutes where they cannot connect to SAS Studio.

However, "kubectl describe" or "kubectl logs" for the sas-formats pod does not show anything.

Can you pls help how to find the root cause for this, and potentially fix it.

the json of the event from Opensearch is below

{
"_index": "viya_logs-viya-prod-eks-ns-2025-08-20",
"_id": "1030066a-9ad1-429f-b8a3-1f52d7363401",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2025-08-20T11:54:58.000Z",
"level": "Warning",
"message": "Readiness probe failed: Get \"http://100.64.14.80:9080/formats/live\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)",
"logsource": "KUBE_EVENT",
"properties": {
"reason": "Unhealthy",
"eventid": "1030066a-9ad1-429f-b8a3-1f52d7363401",
"firsttimestamp": "2025-08-02T05:33:18Z",
"reportingInstance": "ip-10-221-14-200.ca-centra-1.compute.internal",
"object_type": "Pod",
"lastimestamp": "2025-08-20T11:54:58Z",
"count": 5,
"fieldPath": "spec.containers{sas-portable-formats}",
"eventName": "sas-formats-66f6bb44f9-mmnpd.1857dc4bfe06b0b2",
"type": "Warning",
"reportingComponent": "kubelet",
"object": "sas-formats-66f6bb44f9-mmnpd"
},
"kube": {
"pod": "sas-formats-66f6bb44f9-mmnpd",
"namespace": "viya-prod-eks-ns",
"host": "ip-10-220-13-168.ca-central-1.compute.internal",
"pod_id": "1cd18fa0-ec58-4a3c-af6a-4f0f4d6c2581"
}
},
"fields": {
"@timestamp": [
"2025-08-20T11:54:58.000Z"
]
},
"highlight": {
"logsource.keyword": [
"@opensearch-dashboards-highlighted-field@KUBE_EVENT@/opensearch-dashboards-highlighted-field@"
]
},
"sort": [
1755690898000
]
}

Thanks a lot,

Re: SAS formats shows warning and kubernetes and determine couple minutes slowness

gwootton — Fri, 22 Aug 2025 20:05:23 GMT

If the liveness probe fails enough, Kubernetes will restart the pod, so you may need to use kubectl logs --previous to see the logs that include the failure.

Re: SAS formats shows warning and kubernetes and determine couple minutes slowness

drahorg — Mon, 25 Aug 2025 20:33:30 GMT

Thank you Greg for your reply. the SAS Formats pod does not restart. Also both kubectl describe pod and kubectl logs pod does not show any error. I capture this in Opensearch messages showing as error.

Re: SAS formats shows warning and kubernetes and determine couple minutes slowness

gwootton — Tue, 26 Aug 2025 18:27:50 GMT

If the pod did not restart then the liveness probe didn't fail enough to trigger this, suggesting the failure was not ongoing. For sas formats the liveness probe by default is queried every 20 seconds, and if it fails three times the service is restarted.
So it sounds like whatever the issue was, it did not last more than 60 seconds.