topic sas-run-tool: Service executor failed to execute successfully: exit status 1 in Administration and Deployment

sas-run-tool: Service executor failed to execute successfully: exit status 1

drhorg — Wed, 21 Aug 2024 02:20:03 GMT

We have installed Viya 4 Kubernetes, and deployed OpenSearch Dashboards for logging . I see the error

Logsource: sas-run-tool

message : Tool executor failed starting service: exit status 1

Level : Error

then right after

Level : Fatal

Logsource: sas-run-tool

message : Service executor failed to execute successfully: exit status 1

Level : Fatal

Did it happen in your environment and you know what this is? I googled but did not find much for this error.

Thanks a lot,

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

gwootton — Wed, 21 Aug 2024 15:42:21 GMT

What is the name of the pod that is throwing these messages?

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

drhorg — Sat, 31 Aug 2024 02:52:53 GMT

Hi @gwootton , sorry for delay, missed your reply.

kube.pod - sas-backup-purge-job

level -fatal

kube.container - sas-backup-purge-job

kube.annotations.sas_com/component-name

sas-backup-job

Please let me know if any other info needed.

thank you!

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

gwootton — Tue, 03 Sep 2024 14:34:40 GMT

The sas-backup-purge-job I think runs daily and removes old backups, this message suggests one such execution of a job failed. If you look at your job history do you see these show as completed?
kubectl -n namespace get jobs -l app.kubernetes.io/name=sas-backup-purge-job

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

drhorg — Sun, 08 Sep 2024 01:37:23 GMT

Hi @gwootton , yes the purge failed multiple times. Sorry by mistake I accepted my reply as solution instead of your reply. I fixed this now.

thank you for your help!

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Fri, 13 Dec 2024 11:11:59 GMT

Greg,

sometimes my backup gives failed in viya 24.03.

I cannot find the reason, because if I fire this job again, it succeeds.

Maybe you have a solution ?

Regards,

Dik

{"level":"info","version":1,"source":"sas-backupjob","messageKey":"Checking if job is allowed to run.","properties":{"logger":"backupjob","caller":"backupjob/main.go:314"},"timeStamp":"2024-12-13T02:02:04.280263+01:00","message":"Checking if job is allowed to run."}
{"level":"info","version":1,"source":"sas-backupjob","messageKey":"backupjob-log-icu.updating.k8s.job.labels.info.log","messageParameters":{"jobName":"sas-scheduled-backup-dpyxjx3i-28900860"},"properties":{"logger":"backupjob/kubeutils","caller":"kubeutils/kube_client.go:126"},"timeStamp":"2024-12-13T02:02:04.334724+01:00","message":"Updating the Kubernetes job sas-scheduled-backup-dpyxjx3i-28900860 with given label."}
{"level":"info","version":1,"source":"sas-backupjob","messageKey":"backupjob-log-icu.updated.k8s.job.info.log","messageParameters":{"jobName":"sas-scheduled-backup-dpyxjx3i-28900860"},"properties":{"logger":"backupjob/kubeutils","caller":"kubeutils/kube_client.go:152"},"timeStamp":"2024-12-13T02:02:04.345526+01:00","message":"Updated the Kubernetes job the sas-scheduled-backup-dpyxjx3i-28900860."}
{"level":"error","version":1,"source":"sas-backupjob","messageKey":"Job is not allowed to run.","properties":{"logger":"backupjob","caller":"backupjob/main.go:476"},"timeStamp":"2024-12-13T02:02:04.345565+01:00","message":"Job is not allowed to run."}
{"level":"error","version":1,"source":"sas-backupjob","messageKey":"backupjob-log-icu.remote.backup.agent.process.grep.pid.error.log","messageParameters":{"error":"exit status 1"},"properties":{"logger":"backupjob","caller":"backupjob/main.go:529"},"timeStamp":"2024-12-13T02:04:14.495204+01:00","message":"backupjob-log-icu.remote.backup.agent.process.grep.pid.error.log [error:exit status 1]"}
{"level":"error","version":1,"source":"sas-run-tool","messageKey":"Tool executor failed starting service: exit status 1","properties":{"logger":"tools/run","caller":"impl/executor.go:69"},"timeStamp":"2024-12-13T02:04:14.497860+01:00","message":"Tool executor failed starting service: exit status 1"}
{"level":"fatal","version":1,"source":"sas-run-tool","messageKey":"Service executor failed to execute successfully: exit status 1","properties":{"logger":"tools/run","caller":"impl/tooling.go:139"},"timeStamp":"2024-12-13T02:04:14.497901+01:00","message":"Service executor failed to execute successfully: exit status 1"}

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

gwootton — Mon, 16 Dec 2024 14:01:02 GMT

The failure message here is "Job is not allowed to run." Usually this happens because another backup is already running.

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Tue, 17 Dec 2024 11:31:03 GMT

Greg, thank you, but I don't run 2 backups in the night, can I simutlate the check that a backup is already running ?

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

gwootton — Tue, 17 Dec 2024 15:29:00 GMT

From what I can tell keys are created in the SAS Configuration Server with the backup job id (e.g. backup-job-uuid or backup-job-incr-uuid). It checks if such a key is present and, depending on the type of backup being requested will stop if it finds one. It would also stop if it can't create such a key.
You may want to turn on debug logging of the backup, this should provide more detail on the failure. To do that, you could set the environment variable SAS_LOG_LEVEL_BACKUPJOB to debug on the cronjob, for example:
kubectl -n namespace set env cronjob/sas-scheduled-backup-job SAS_LOG_LEVEL_BACKUPJOB=DEBUG

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Thu, 19 Dec 2024 07:46:46 GMT

Greg, thank you, now we have more output, we will investigate further..

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Thu, 19 Dec 2024 09:13:12 GMT

Greg, we found it.

the sas-backup-purge-job cannot run simultaneous with the sas-scheduled-backup-job.

So this explains why it went ok if we run it by hand in the morning.

We will reschedule the full backups.

kubectl -n $ns get cronjob | grep backup | grep False
sas-backup-purge-job 15 0 1/1 * ? False 0 8h 127d
sas-scheduled-backup-job 0 1 * * 0 False 0 4d8h 127d
sas-scheduled-backup-kbx7pj33 0 1 * * 1-5 False 0 8h 16d

Adjusted to :
sas-backup-purge-job 15 0 1/1 * ? False 0 8h 127d
sas-scheduled-backup-job 0 4 * * 0 False 0 4d8h 127d
sas-scheduled-backup-kbx7pj33 0 4 * * 1-5 False 0 8h 16d

Tomorrow we know it for sure.

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Tue, 07 Jan 2025 14:56:44 GMT

Greg,

the reason is that the purge job runs for hours .

The pg_dump must be dumping with -Fd , we get 469000 *.dat.gz files . The purge proces has a lot of work to do with so many files.

Is it possible to adjust this to -Fc just one bigfile, or some other solution ...

Regards

Dik

We are on 2403 lts , maybe this behavior will change in the next upgrade.

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

gwootton — Thu, 09 Jan 2025 21:33:28 GMT

You are correct that the backup uses the --format directory option for pgdump, this is hard-coded and not customizable.
As far as how long purge is taking to complete though, that seems high. My oldest backup's postgres dump contains 36,544 files and my last purge job took 2 minutes to complete. In the logs for the purge job do you see any indication of why it's taking so long?

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Tue, 21 Jan 2025 14:57:02 GMT

We see entries in the table pg_catalog.pg_largeobject, for example job history and report transfer data. We export the reports with the sas-viya cli tool.

These entries are exported as blob<loid>.dat.gz
- How can I delete the job history in viya ?

- For the reports we are investigating.

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Mon, 27 Jan 2025 13:33:28 GMT

Greg, more information :

I made a sas studio job ;

cas;
%put "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" ;

run ;

Logged into postgres pod : kn exec sas-crunchy-platform-postgres-00-b56g-0 -it -- sh

psql

\c SharedServices

SharedServices=# select loid from pg_catalog.pg_largeobject where data like '%ZZZZZ%' ;
loid
---------
6612731
6612732
6613030
6613031
6613039
6613038
6765563
(7 rows)
What is in the data column ?

select loid,substr(encode(data, 'escape'),1100) as data from pg_catalog.pg_largeobject where data like '%ZZZZZ%' limit 1;

loid | data
---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6612731 | kers."},{"version":1,"type":"source","line":"593 %put \\"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ\\" ;"},{"version":1,"type":"normal","line":"\\"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ\\""},{"version":1,"type":"source","line":"594 "},{"version":1,"type":"source","line":"595 run ;"},{"version":1,"type":"source","line":"596 %JESEND"},{"version":1,"type":"note","line":""}],"count":674,"limit":674,"links":[]}
(1 row)

Login the backup pod OR another pod which has the backup directory mounted and goto the directory where the pgdump file is and zcat the file blob_<loid>.dat.gz
/sasviyabackup/20250127-010206F/__default__/postgres/SharedServices_pg_dump.dmp $ zcat blob_6612731.dat.gz | tail -40 | fold -80 | tail -10
The user is <d.pater> and the active caslib is CASUSER(<d.pater>)
."},{"version":1,"type":"note","line":"NOTE: The SAS option SESSREF was updated
with the value CASAUTO."},{"version":1,"type":"note","line":"NOTE: The SAS macro
_SESSREF_ was updated with the value CASAUTO."},{"version":1,"type":"note","lin
e":"NOTE: The session is using 2 workers."},{"version":1,"type":"source","line":
"593 %put \"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ\" ;"},{"version":
1,"type":"normal","line":"\"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ\""
},{"version":1,"type":"source","line":"594 "},{"version":1,"type":"source","lin
e":"595 run ;"},{"version":1,"type":"source","line":"596 %JESEND"},{"version":
1,"type":"note","line":""}],"count":674,"limit":674,"links":[]}

Questions :

1.is this normal behavior ?

2. Must i register a case with SAS for this behavior ?

Regards

Dik

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

gwootton — Mon, 27 Jan 2025 14:03:56 GMT

When a file gets upload to the files service, the data is physically stored in a large object in postgres.

This includes files uploaded through SAS Content, but also for any job that has run, the results of that job.

This job history, including logs, is preserved based on the expiration attribute of the job, which is set based on the "expiresAfter" parameter of the jobRequest.

When you create a jobRequest through SAS Studio, this expiration is set to 72 hours. When you create a job in Environment Manager, the expiration is not set, meaning the job history is preserved indefinitely.

I created a series of pyviyatools tools to address the issue of job history being preserved indefinitely:
1. setjobrequestexpire.py - https://github.com/sassoftware/pyviyatools/blob/master/setjobrequestexpire.py - Allows you to set the expiresAfter parameter for jobRequests. This will cause any future jobs run under that jobRequest to carry an expiration based on that parameter.
2. deletejobhistory.py - https://github.com/sassoftware/pyviyatools/blob/master/deletejobhistory.py - Allows you to delete the job history (contents of the Monitoring tab) based on filters, so for example any jobs that completed over a week ago, or all the history for a specific job.
3. deleteorphanedfiles.py - https://github.com/sassoftware/pyviyatools/blob/master/deleteorphanedfiles.py - When you delete a job history object, the associated files for that object should be deleted as well automatically. Sometimes this process fails resulting in a file object without a valid parent (as the job history object has been deleted). This tool identifies any files objects without a parent (i.e. the parentUri when called returns a 404), and deletes them.

So #1 prevents future job history from being preserved indefinitely and #2 and #3 clean up existing history.

For exports, the transfer service has functions to create an export package and download it. After you've downloaded it, the package is still present on the server and could be downloaded again, until such time as you delete the export package. When I'm doing an export, I include a delete to keep the database free of old packages once they've been downloaded:
sas-viya transfer export (produces ID as output)
sas-viya transfer download --id id-from-above
sas-viya transfer delete --id-from-above

You can also remove old packages using the sas-viya transfer delete CLI command if you no longer need them on the server, the sas-viya transfer list command will show you the packages that still exist on the server.

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Tue, 28 Jan 2025 10:21:36 GMT

Mr.Gwooton,

I had some problems with the quoting 😉

This works for the Update CAS Audit Data, we have a lot of records, I also could have used createdBy,sas.audit.

python3 deletejobhistory.py -e -f 'eq(jobRequest.name,"Update CAS Audit data")' -p 1000 -l 100 -w

Thank you, we will add 72 hours or so, to our jobs that don't have an expiry set.

I will also take a look at the orphan script.

Regards

Dik

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Wed, 29 Jan 2025 11:56:43 GMT

Mr. Gwooton

I still have some job related entries for example :

select count(*) from workloadorchestrator.job_std_output ;

count

---------

4495370

(1 row)

SharedServices=# select count(*) from workloadorchestrator.job_std_output where line_txt like '%App.ComputeServer.Communication.HTTP%';

count

---------

4266126

(1 row)

SharedServices=# SELECT pg_size_pretty(pg_total_relation_size('workloadorchestrator.job_std_output')) AS table_size;

table_size

------------

3118 MB

(1 row)

Do you know how we can also get rid of these entries ?

I cannot find a logger with the name App.ComputeServer.Communication.HTTP

sas-viya -p $ns workload-orchestrator logs levels get | grep Loggers |awk -F"name:" '{for (i = 1; i <= NF; i++) {print $i}}'| grep -v NULL
Loggers [map[level:INFO
App.NLS] map[level:ERROR
App.tk.HTTPC.Conn] map[level:FATAL
App.tk.HTTPC.State] map[level:FATAL
App.tk.HTTPC.debug] map[level:FATAL
App.tk.jwtkeystore] map[level:INFO
App.tk.HTTPC.DNSCache] map[level:FATAL
App.tk.eam.rsa.Buffer] map[level:FATAL
App.tk.eam.ssl.openssl111] map[level:ERROR
App.Grid.SGMG.Log.Config.JSON] map[level:FATAL
App.SQLServices.RTE.Statement] map[level:FATAL
App.SQLServices.RTE.Environment] map[level:FATAL
App.SQLServices.RTE.Connection.DCM] map[level:FATAL
App.SQLServices.RTE.Statement.POSTGRES.stats]]

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

gwootton — Wed, 29 Jan 2025 14:34:51 GMT

These records are not for jobs from the jobExecution service (those that show in the Monitoring tab of the Jobs and Flows page in SAS Environment Manager) but records stored by SAS Workload Orchestrator related to its job history. This would include anything launched to SWO, so would include all compute server, connect server and batch servers that result in a pod being created. The table you are looking at, job_std_output, captures a row for each line returned to stdout from each job up to the maxStdOutLines configuration in SWO.

Removing these entries would remove the job history from the Workload Orchestrator page in Environment Manager. There is no function within SWO to remove these entries, so if you wanted to remove old entries from this table you would need to manipulate the tables in the database directly rather than through any interface Viya provides.

If you engage SAS Technical Support, they can provide you with a script you can run on a SAS Workload Orchestrator StatefulSet pod to remove old entries.

The script accepts as arguments a retention period in days and limit in the total number of job records to process, and it builds and executes a SQL query to:

Build a temporary table of job IDs that ended prior to the retention period up to the limit provided from the main job table.
For each of those jobs, delete any associated records in the supporting tables.
For each of those jobs, delete the record in the main job table.
VACUUM ANALYZE and REINDEX each table.

Because this can take a long time to finish, the script nohup's the SQL query so you don't need to remain connected to the pod while it runs.

Re: sas-run-tool: Service executor failed to execute successfully: exit status 1

paterd2 — Thu, 30 Jan 2025 08:30:18 GMT

Thank you, with this thread and the information you gave me , we are more in control.