Launch of Server SASApp -Workspace server for user failed

[Madhan_cog1]

Hi Team,

We have SAS Grid 9.4m7 version deployed in Redhat8.1 and the topology is 1 control 2 compute , 3 metadata and 2 Midtier.

One of the user is getting the below issue while using SAS EG.

User was able to execute the program yesterday.

We could see only this in Object Spawner Logs

2022-09-21T13:42:54,383 INFO [00508933] :sasprd - Client connection 41693 for user denise17345 closed.
2022-09-21T13:43:57,030 INFO [00508976] :denise17345 - New client connection (40935) accepted from server port 8591 for user denise17345. Encryption level is Credentials using encryption algorithm SASProprietary. Peer IP address and port are [::ffff:10.17.144.82]:54316 for APPNAME=SAS Enterprise Guide.
2022-09-21T13:43:57,077 INFO [00508976] :denise17345 - New out call client connection (42086) for user sastrust@saspw. Encryption level is Credentials using encryption algorithm SASPROPRIETARY. Peer IP address and port are [10.24.16.146]:8561.

Kindly let us know what could be the issue 

Thanks,

Madhan M

[Madhan_cog1]

Can someone Reply ASAP

[Madhan_cog1]

2022-09-21T15:17:59,487 ERROR [00346142] :sasprd - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'

2022-09-21T15:17:59,574 ERROR [00351814] :sasprd - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/state'.

2022-09-21T15:17:59,574 ERROR [00351814] :sasprd - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'

2022-09-21T15:17:59,732 ERROR [00352142] :sasprd - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/state'.

2022-09-21T15:17:59,732 ERROR [00352142] :sasprd - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'

[sasprd@J1IADSASCTL01 Logs]$ pwd

/opt/sas/gridconf/Lev1/Logs

[SASKiwi]

If you require a quick response then please open a track with SAS Tech Support. No service level targets apply to the SAS Communities. Is only this user affected or are all users affected? If all users are affected, start by checking that all SAS server services are up and running.

[gwootton]

This error looks to be related to the user not having credentials to log on to SAS Workload Orchestrator. So the user does not have a password stored in Metadata or in an authinfo file, or in the case of negotiate authentication a kerberos token is not available.

Configuring SAS Workload Orchestrator Authentication
https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/gridref/p0vse60qjpoxn6n1rrllbkqd2rzk.htm

[Madhan_cog1]

Hi All,

The user is able to launch Workspace server with there id from the terminal but unable to use SAS EG

Am attaching the logs from EG after enabling the logging . see if this tells you something.

Also we tried to  test the connection from SAS Integration Technolgies and below is the output

See if someone can suggest anything with this log.

Thanks,

Madhan M

[Madhan_cog1]

2022-09-22T15:49:42,002 INFO [00412132] :srujana40767 - Grid credentials retrieved from authdomain 'DefaultAuth' in metadata.
2022-09-22T15:49:42,003 INFO [00000010] :sasprd - Client connection 7253 for user sastrust@saspw closed.
2022-09-22T15:49:42,091 ERROR [00412132] :srujana40767 - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/info'.
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - PROVIDER(SAS): Exception occurred: HTTP Status:170, Error:0xaa, Msg:'HTTP/1.1 403 Forbidden', Details:'<none>'
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - Invalid argument
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - The launch of server SASApp - Workspace Server for user srujana40767 failed.
2022-09-22T15:49:42,092 ERROR [00412132] :srujana40767 - The specified uuid E6371350-65B5-8044-9F73-1FFD80D5CFB2 did not match any process managed by this spawner.
2022-09-22T15:49:42,092 INFO [00412132] :sasprd - Client connection 7934 for user srujana40767 closed.
2022-09-22T15:49:44,102 ERROR [00316533] :sasprd - The requested operation is not permitted for URL 'https://j1iadsasctl01.mocr-nt1.otsuka.com:8901/sasgrid/api/jobs/state'.

[gwootton]

Have you engaged technical support? You're getting a HTTP 403 response which means SAS Workload Orchestrator is not authenticating your user. It's logs may have more detail.

[jwilson]

I'm having the same exact problem.  How long have you had your GRID server?  Ours is brand new.  I think what's happening to us (this may or may not be what's happening with you) is that our main grid manager went down for a second, and our backup manager came up but might not be properly configured. 

[jwilson]

@Madhan_cog1were you able to solve this issue?  If so can you please post an update?  I'm struggling through the same exact issue.

Thank you!

Jonathan

[SASKiwi]

@jwilson  - Have you opened a Tech Support track yet as they are in the best position to help you.

[jwilson]

Yes.

[SASKiwi]

@jwilson - Are all SAS users affected by this error or just some? If it is just some then I'd compare OS / metadata permissions between those that work and those that don't. If it is all users, then it is more likely that a SAS mid-tier server service is not working properly. Is this a new installation and hasn't worked correctly so far or was it working OK but now it isn't? If it is the latter, then one option would be to reboot all of your SAS servers in the correct sequence to see if that fixes the problem.

[jwilson]

@SASKiwithanks for the suggestion.  It's only some users.  I've been trying to look for differences between those that work and those that don't but I think I might be looking at the wrong place.  It's a brand new system.  I think the credentials are coming from AD but the server itself is Linux based.  We tried rebuilding the affected users' accounts but that didn't do it. 

[gwootton]

If you are also getting a 403 response from workload orchestrator in the Object Spawner log, I would suggest you look in the Workload Orchestrator log which may have more detail as to why the authentication is failing. Depending on how authentication is configured, workload orchestrator either authenticates against the host or using Kerberos.