topic Re: Launch of Server SASApp -Workspace server for user failed in Administration and Deployment

Launch of Server SASApp -Workspace server for user failed

Madhan_cog1 — Wed, 21 Sep 2022 18:32:15 GMT

Hi Team,

We have SAS Grid 9.4m7 version deployed in Redhat8.1 and the topology is 1 control 2 compute , 3 metadata and 2 Midtier.

One of the user is getting the below issue while using SAS EG.

User was able to execute the program yesterday.

We could see only this in Object Spawner Logs

2022-09-21T13:42:54,383 INFO [00508933] :sasprd - Client connection 41693 for user denise17345 closed.
2022-09-21T13:43:57,030 INFO [00508976] :denise17345 - New client connection (40935) accepted from server port 8591 for user denise17345. Encryption level is Credentials using encryption algorithm SASProprietary. Peer IP address and port are [::ffff:10.17.144.82]:54316 for APPNAME=SAS Enterprise Guide.
2022-09-21T13:43:57,077 INFO [00508976] :denise17345 - New out call client connection (42086) for user sastrust@saspw. Encryption level is Credentials using encryption algorithm SASPROPRIETARY. Peer IP address and port are [10.24.16.146]:8561.

Kindly let us know what could be the issue

Thanks,

Madhan M

Re: Launch of Server SASApp -Workspace server for user failed

Madhan_cog1 — Wed, 21 Sep 2022 19:08:54 GMT

Can someone Reply ASAP

Re: Launch of Server SASApp -Workspace server for user failed

Madhan_cog1 — Wed, 21 Sep 2022 19:36:56 GMT

2022-09-21T15:17:59,487 ERROR [00346142] :sasprd - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'

2022-09-21T15:17:59,574 ERROR [00351814] :sasprd - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/state'.

2022-09-21T15:17:59,574 ERROR [00351814] :sasprd - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'

2022-09-21T15:17:59,732 ERROR [00352142] :sasprd - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/state'.

2022-09-21T15:17:59,732 ERROR [00352142] :sasprd - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'

[sasprd@J1IADSASCTL01 Logs]$ pwd

/opt/sas/gridconf/Lev1/Logs

Re: Launch of Server SASApp -Workspace server for user failed

SASKiwi — Wed, 21 Sep 2022 20:03:54 GMT

If you require a quick response then please open a track with SAS Tech Support. No service level targets apply to the SAS Communities. Is only this user affected or are all users affected? If all users are affected, start by checking that all SAS server services are up and running.

Re: Launch of Server SASApp -Workspace server for user failed

gwootton — Wed, 21 Sep 2022 20:36:12 GMT

This error looks to be related to the user not having credentials to log on to SAS Workload Orchestrator. So the user does not have a password stored in Metadata or in an authinfo file, or in the case of negotiate authentication a kerberos token is not available.

Configuring SAS Workload Orchestrator Authentication
https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/gridref/p0vse60qjpoxn6n1rrllbkqd2rzk.htm

Re: Launch of Server SASApp -Workspace server for user failed

Madhan_cog1 — Thu, 22 Sep 2022 16:19:45 GMT

Hi All,

The user is able to launch Workspace server with there id from the terminal but unable to use SAS EG

Am attaching the logs from EG after enabling the logging . see if this tells you something.

Also we tried to test the connection from SAS Integration Technolgies and below is the output

See if someone can suggest anything with this log.

Thanks,

Madhan M

Re: Launch of Server SASApp -Workspace server for user failed

Madhan_cog1 — Thu, 22 Sep 2022 20:00:10 GMT

2022-09-22T15:49:42,002 INFO [00412132] :srujana40767 - Grid credentials retrieved from authdomain 'DefaultAuth' in metadata.
2022-09-22T15:49:42,003 INFO [00000010] :sasprd - Client connection 7253 for user sastrust@saspw closed.
2022-09-22T15:49:42,091 ERROR [00412132] :srujana40767 - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/info'.
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - Invalid argument
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - The launch of server SASApp - Workspace Server for user srujana40767 failed.
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - The specified uuid E6371350-65B5-8044-9F73-1FFD80D5CFB2 did not match any process managed by this spawner.
2022-09-22T15:49:42,092 INFO [00412132] :sasprd - Client connection 7934 for user srujana40767 closed.
2022-09-22T15:49:44,102 ERROR [00316533] :sasprd - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/state'.

Re: Launch of Server SASApp -Workspace server for user failed

gwootton — Thu, 22 Sep 2022 20:18:38 GMT

Have you engaged technical support? You're getting a HTTP 403 response which means SAS Workload Orchestrator is not authenticating your user. It's logs may have more detail.

Re: Launch of Server SASApp -Workspace server for user failed

jwilson — Wed, 28 Sep 2022 23:43:58 GMT

I'm having the same exact problem. How long have you had your GRID server? Ours is brand new. I think what's happening to us (this may or may not be what's happening with you) is that our main grid manager went down for a second, and our backup manager came up but might not be properly configured.

Re: Launch of Server SASApp -Workspace server for user failed

jwilson — Tue, 04 Oct 2022 00:06:16 GMT

@Madhan_cog1were you able to solve this issue? If so can you please post an update? I'm struggling through the same exact issue.

Thank you!

Jonathan

Re: Launch of Server SASApp -Workspace server for user failed

SASKiwi — Tue, 04 Oct 2022 00:36:40 GMT

@jwilson - Have you opened a Tech Support track yet as they are in the best position to help you.

Re: Launch of Server SASApp -Workspace server for user failed

jwilson — Tue, 04 Oct 2022 00:37:53 GMT

Yes.

Re: Launch of Server SASApp -Workspace server for user failed

SASKiwi — Tue, 04 Oct 2022 01:22:28 GMT

@jwilson - Are all SAS users affected by this error or just some? If it is just some then I'd compare OS / metadata permissions between those that work and those that don't. If it is all users, then it is more likely that a SAS mid-tier server service is not working properly. Is this a new installation and hasn't worked correctly so far or was it working OK but now it isn't? If it is the latter, then one option would be to reboot all of your SAS servers in the correct sequence to see if that fixes the problem.

Re: Launch of Server SASApp -Workspace server for user failed

jwilson — Tue, 04 Oct 2022 01:32:28 GMT

@SASKiwithanks for the suggestion. It's only some users. I've been trying to look for differences between those that work and those that don't but I think I might be looking at the wrong place. It's a brand new system. I think the credentials are coming from AD but the server itself is Linux based. We tried rebuilding the affected users' accounts but that didn't do it.

Re: Launch of Server SASApp -Workspace server for user failed

gwootton — Tue, 04 Oct 2022 13:15:08 GMT

If you are also getting a 403 response from workload orchestrator in the Object Spawner log, I would suggest you look in the Workload Orchestrator log which may have more detail as to why the authentication is failing. Depending on how authentication is configured, workload orchestrator either authenticates against the host or using Kerberos.