Object Spawner and SAS/SHARE Servers keep restarting on their own

[Abhijit]

Hi there

In one of our SAS environments, Object Spawner and SAS/SHARE Servers keep restarting on their own. When SAS was installed, 2 workspace servers were installed and loading balancing was put in place. Because load balancing was not working properly, we disabled the load balancing through the SAS Metadata Server.

Please shed some light on it if you can.

Excerpt from the Object Spawner log:

20160504:09.30.08.04: 00000004:NOTE: AUTOEXEC processing beginning; file is /usr/local/sas/Centric/Lev1/SASCentric/WorkspaceServer/autoexec.sas.
20160504:09.30.08.04: 00000004:
20160504:09.30.08.11: 00000004:NOTE: Libref TRANS was successfully assigned as follows:
20160504:09.30.08.11: 00000004:NOTE: Engine:        META
20160504:09.30.08.11: 00000004:NOTE: Physical Name: /usr/local/sas/Centric/Lev1/SASCentric/Data/Trans
20160504:09.30.08.16: 00000004:ERROR: Attempt to connect to server  failed.
20160504:09.30.08.16: 00000004:ERROR: A communication subsystem partner link setup request failure has occurred.
20160504:09.30.08.16: 00000004:ERROR: The specified address is not available.
20160504:09.30.08.16: 00000004:ERROR: Error in the LIBNAME statement.
20160504:09.30.08.39: 00000004:NOTE: Libref CCDM was successfully assigned as follows:
20160504:09.30.08.39: 00000004:NOTE: Engine:        META
20160504:09.30.08.39: 00000004:NOTE: Physical Name: PROTO
20160504:09.30.08.64: 00000004:NOTE: Libref CDL was successfully assigned as follows:
20160504:09.30.08.64: 00000004:NOTE: Engine:        META
20160504:09.30.08.64: 00000004:NOTE: Physical Name: PROTO
20160504:09.30.08.70: 00000004:ERROR: Attempt to connect to server "51.127.149.53".__8551 failed.
20160504:09.30.08.70: 00000004:ERROR: A communication subsystem partner link setup request failure has occurred.
20160504:09.30.08.70: 00000004:ERROR: Communication request rejected by partner: security verification failure.
20160504:09.30.08.70: 00000004:ERROR: Error in the LIBNAME statement.
20160504:09.30.08.72: 00000004:NOTE: Libref UAT was successfully assigned as follows:
20160504:09.30.08.72: 00000004:NOTE: Engine:        META
20160504:09.30.08.72: 00000004:NOTE: Physical Name: /usr/local/sas/Centric/Lev1/SASCentric/Data/UAT
20160504:09.30.08.78: 00000004:ERROR: Attempt to connect to server "51.127.149.53".__8551 failed.
20160504:09.30.08.78: 00000004:ERROR: A communication subsystem partner link setup request failure has occurred.
20160504:09.30.08.78: 00000004:ERROR: Communication request rejected by partner: security verification failure.
20160504:09.30.08.78: 00000004:ERROR: Error in the LIBNAME statement.

Excerpts from the SAS/SHARE Server log:

04MAY2016:09:29:39.113 Server library METADATA ('/usr/local/sas/Centric/Lev1/SASCentric/Data/MetaData' V9) accessed as METADATA by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:39.176 User u5271390(33) has created "METAREB "(3) under "Object Server"(1).
04MAY2016:09:29:39.186 Server library CONTROL ('/usr/local/sas/Centric/Lev1/SASCentric/Data/Control' V9) accessed as CONTROL by "METAREB "(3) of user u5271390(33).
04MAY2016:09:29:39.595 METADATA.CONFIG_TABLE.DATA(1) opened for input/S via engine V9 by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:39.801 METADATA.CONFIG_TABLE.DATA(1) closed by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:39.867 METADATA.CONFIG_TABLE.DATA(2) opened for input/S via engine V9 by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:40.058 METADATA.CONFIG_TABLE.DATA(2) closed by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:40.121 METADATA.CONFIG_TABLE.DATA(3) opened for input/S via engine V9 by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:40.321 METADATA.CONFIG_TABLE.DATA(3) closed by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:40.385 METADATA.CONFIG_TABLE.DATA(4) opened for input/S via engine V9 by "METAREB "(2) of user u5271390(33).
04MAY2016:09:29:40.582 METADATA.CONFIG_TABLE.DATA(4) closed by "METAREB "(2) of user u5271390(33).
ERROR: Access denied.
ERROR: Access denied.
NOTE: SAH219001I SAS/SHARE Server __8551, State, stopping
04MAY2016:09:30:35.895 Normal termination of SAS server __8551 has occurred.
04MAY2016:09:30:35.896 User u5271390(33) has terminated "METAREB "(3) (under "Object Server"(1)).

Thanks

Abhi

[Mark_sas]

I think you're going to have to get Tech Support involved.  The symptoms appear to be fairly specific with the Object Spawner apparently having difficulty communicating with the SHARE Server.  On the bright side, Tech Support is pretty good at diagnosing reproducible behavior.

[GyaniBaba]

1. which sas version are you using?
2.  can you run SASHFADD and check for any critical hot fixes not installled.
3. security verification failure? what user name and password are being used for share server? check shareserver.sas or check share server property in sas mc. are those user/password correct?
4. how it is happening, can you list down the sequence of activities? Is it possible to share full log along with metadata log with issue timestamp?