Unhandled exception in Scheduling service

[viswmmo]

Hi!

I am getting error messages in the schedule servIce log, but can not find the job that is causing them. The messages are popping up once per hour telling me the job does not exist:

2022-02-25 03:05:00.105 INFO 6009 --- [ SASSched-4] org.quartz.core.JobRunShell : service [0437c62e08d2f0a4]
Job /jobExecution/jobRequests/b3ddd3c3-54a8-4801-8c26-b487eb901cca/jobs.55ad3b9c-a01f-4d9d-95e2-5e4b5e8446b1 threw a JobExecu
tionException:

org.quartz.JobExecutionException: Scheduling Service job execution resulted in unhandled exception
at com.sas.scheduling.quartz.JobWrapper.executeInternal(JobWrapper.java:321)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : [{"errorCode":0,"message":"Job request not
found with id: b3ddd3c3-54a8-4801-8c26-b487eb901cca","details":["traceId: 6fed448ee36d6930","path: /jobExecution/jobRequests/b
3ddd3c3-54a8-4801-8c26-b487eb901cca/jobs"],"links":[],"version":2,"httpStatusCode":404}]
at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:113)

Job is not listed in Schedule service:

./sas-admin job requests show --id b3ddd3c3-54a8-4801-8c26-b487eb901cca 

The following errors have occurred:

Job request not found with id: b3ddd3c3-54a8-4801-8c26-b487eb901cca

Can someone please give me a clue on where to find and delete this mysterious job?

Best Regards

Mattias

Error message in schedule service:

2022-02-25 03:05:00.105 INFO 6009 --- [ SASSched-4] org.quartz.core.JobRunShell : service [0437c62e08d2f0a4]
Job /jobExecution/jobRequests/b3ddd3c3-54a8-4801-8c26-b487eb901cca/jobs.55ad3b9c-a01f-4d9d-95e2-5e4b5e8446b1 threw a JobExecu
tionException:

org.quartz.JobExecutionException: Scheduling Service job execution resulted in unhandled exception
at com.sas.scheduling.quartz.JobWrapper.executeInternal(JobWrapper.java:321)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : [{"errorCode":0,"message":"Job request not
found with id: b3ddd3c3-54a8-4801-8c26-b487eb901cca","details":["traceId: 6fed448ee36d6930","path: /jobExecution/jobRequests/b
3ddd3c3-54a8-4801-8c26-b487eb901cca/jobs"],"links":[],"version":2,"httpStatusCode":404}]
at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:113)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:184)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:672)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:581)
at com.sas.scheduling.quartz.JobWrapper.lambda$executeInternal$1(JobWrapper.java:293)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

[gwootton · Accepted solution]

It looks like you have a job scheduled that no longer exists, so when you do the sas-admin job requests show command it, like scheduler, can't find the job.
You would instead need to query scheduler, i.e. http(s)://webhost:port/scheduler/jobs to find the schedule job with that associated jobRequest. We can use a filter to narrow it down, for example:

http(s)://webhost:port/scheduler/jobs?filter=contains(request.uri,"jobRequests/b3ddd3c3-54a8-4801-8c26-b487eb901cca")

This should return any scheduler jobs that have that associated request. The response should also contain a "DELETE" link you can use to the delete the schedule. You could use curl and jq to call and read these.

# Define which profile you are using for sas-admin
profile=viya
# Login with it
sas-admin --profile $profile auth login
# Pull the token into a variable
token=$(jq ."${profile}" ~/.sas/credentials.json | jq -r '."access-token"')
# Pull the base URL into a variable
baseurl=$(jq .${profile} ~/.sas/config.json | jq -r '."sas-endpoint"')
# Call the /scheduler/jobs URI to find the problem job, returning the name(s)
curl -s "${baseurl}/scheduler/jobs?filter=contains(request.uri,'jobRequests/b3ddd3c3-54a8-4801-8c26-b487eb901cca')" -H "Authorization: Bearer $token" | jq -r '.items[].name'
# Get the URIs to delete it/them:
curl -s "${baseurl}/scheduler/jobs?filter=contains(request.uri,'jobRequests/b3ddd3c3-54a8-4801-8c26-b487eb901cca')" -H "Authorization: Bearer $token" | jq -r '.items[].links[] | select( .rel == "delete")|.href'

You could then use a curl -X DELETE ${baseurl}/{{ URI }} -H "Authorization: Bearer $token" to delete the schedule(s).