"sas-files" pod fail with OOM in Viya 4

[drhorg]

in Viya 4 2024.09 LTS on AWS EKS I get sporadic "sas-files" pod failure due to OOM. While running the "Kubectl get logs" for the previous sas-files pod (to check the failure reason) :

"{"version":1,"timeStamp":"2025-03-14T13:28:12.117Z","level":"warn","source":"sas-files","message":"Reached the maximum number of URI tags for 'http.client.requests'. Are you using 'uriVariables'?","properties":{"logger":"org.springframework.boot.actuate.autoconfigure.metrics.OnlyOnceLoggingDenyMeterFilter","thread":"https-jsse-nio-8080-exec-4","__session":"88884b07-ff5f-49e5-bea6-49d1b1cea81f","username":"John.Smith@somesascompany.com"}}
At this point it hits the OOM . then pod logs additional entries : 

{"version":1,"timeStamp":"2025-03-14T18:16:58.564Z","level":"error","source":"sas-files","message":"com.sas.commons.rest.exceptions.ResourceException: An error occurred. The request failed.","properties":{"logger":"com.sas.commons.rest.ExceptionLog","thread":"https-jsse-nio-8080-exec-7","username":"sas.catalog"},"messageParameters":{"0":"com.sas.commons.rest.exceptions.ResourceException","1":"An error occurred. The request failed."}}
{"version":1,"timeStamp":"2025-03-14T18:22:22.522Z","level":"error","source":"sas-files","message":"ERROR: large object 3607475 does not exist Exception : org.postgresql.util.PSQLException: ERROR: large object 3607475 does not exist","properties":{"logger":"com.sas.svcs.file.domain.PostgresContentDAOImpl","thread":"https-jsse-nio-8080-exec-3","username":"sas.searchIndex"}}
{"version":1,"timeStamp":"2025-03-14T18:22:22.522Z","level":"error","source":"sas-files","message":"An error occurred. The request failed. Exception : ResourceException: An error occurred. The request failed. Http Status: 500 INTERNAL_SERVER_ERROR","properties":{"logger":"com.sas.svcs.file.domain.PostgresContentDAOImpl","thread":"https-jsse-nio-8080-exec-3","username":"sas.searchIndex"}}
{"version":1,"timeStamp":"2025-03-14T18:22:22.531Z","level":"error","source":"sas-files","message":"com.sas.commons.rest.exceptions.ResourceException: An error occurred. The request failed.","properties":{"logger":"com.sas.commons.rest.ExceptionLog","thread":"https-jsse-nio-8080-exec-3","username":"sas.searchIndex"},"messageParameters":{"0":"com.sas.commons.rest.exceptions.ResourceException","1":"An error occurred. The request failed."}}
{"version":1,"timeStamp":"2025-03-14T18:22:22.55Z","level":"error","source":"sas-files","message":"ERROR: large object 3607475 does not exist Exception : org.postgresql.util.PSQLException: ERROR: large object 3607475 does not exist","properties":{"logger":"com.sas.svcs.file.domain.PostgresContentDAOImpl","thread":"https-jsse-nio-8080-exec-1","username":"sas.catalog"}}
{"version":1,"timeStamp":"2025-03-14T18:22:22.551Z","level":"error","source":"sas-files","message":"An error occurred. The request failed. Exception : ResourceException: An error occurred. The request failed. Http Status: 500 INTERNAL_SERVER_ERROR","properties":{"logger":"com.sas.svcs.file.domain.PostgresContentDAOImpl","thread":"https-jsse-nio-8080-exec-1","username":"sas.catalog"}}
{"version":1,"timeStamp":"2025-03-14T18:22:22.552Z","level":"error","source":"sas-files","message":"com.sas.commons.rest.exceptions.ResourceException: An error occurred. The request failed.","properties":{"logger":"com.sas.commons.rest.ExceptionLog","thread":"https-jsse-nio-8080-exec-1","username":"sas.catalog"},"messageParameters":{"0":"com.sas.commons.rest.exceptions.ResourceException","1":"An error occurred. The request failed."}}

One simple solution would be to increase the pod/deploy memory, but I assume this user might generate too high memory but I am not clear of the process, can you please help to troubleshoot for the root cause? 

Thanks a lot!

[gwootton · Accepted solution]

I think increasing the memory limit for your files service would be an appropriate step, you could add reference in the transformers: section of your kustomization.yaml to do this, with the file having the following contents (the default is 1Gi):
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: sas-files-limits-patch
patch: |-
- op: replace
path: /spec/template/spec/containers/0/resources/limits/memory
value: 2Gi
target:
name: sas-files
kind: Deployment

The files service stores file content in the SAS Infrastructure Data Server as large objects, as well as file metadata like the file ID, content ID, file name, etc in it's database tables.

The error message "ERROR: large object ####### does not exist" suggests the files service created the metadata for a file object in the database, but the actual file contents are not present in the database. This could be an artifact of the pod being killed during the file storage process.
If a process requests the file content (/files/files/{file_id}/content endpoint), the files service will attempt to retrieve those contents and if the large object is not present, the error above will occur. In this case, the requests for the contents appear to be coming from sas.catalog and sas.searchIndex.

Usually these errors don't cause any problems, except that the file content is not available and the request for that content would be returned an HTTP 500 response.

[drhorg]

Thanks a lot Greg for the in-depth and clear reply!