Error in Deploying Middle Tier in SAS9.4 M7 . Bad response from object type Collection.DM (6005)

[alko13]

Hello,

We are doing an installation/configuration of IFRS9 over Risk Stratum.

This is a 3 server setup

• Metadata
• Compute
• Middle Tier

OS is Redhat Enterprise Linux 8.6 (Ootpa)

Installation and Configuration has been successful in Metadata and Compute.

However an issue is encountered while deploying the Middle Tier Server on Stage 12

Load Content for the Web Infrastructure Platform with below error message:

As noticed on the SAS WIP Services Log in Lev1/Web/Logs/SASServer1_1 and the Web App Server Log of SASServer1 it is outputting these type of messages:

java.net.BindException: Address already in use (Bind Failed)

Any suggestions are appreciated.

Thanks!

[alko13]

Another observation from the log:

webinfpltfm_loadContent_2023-01-11-15.16.53.log

Somehow the failure begins once it processes objecttype collection 6005. But prior to this seems everything is fine:

[createObjectTypes.groovy]      [echo] Processing object type Term.BDN (6003)

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Opening connection to https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6003

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Will create object type at https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Opening connection to https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting icon for type: 6003 at url https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6003/ui/icon

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting localized names for type: 6003 at url https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6003/ui/localizedNames

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Processing object type ArchitectJob.DM (6004)

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Opening connection to https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6004

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Will create object type at https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Opening connection to https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting icon for type: 6004 at url https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6004/ui/icon

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting localized names for type: 6004 at url https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6004/ui/localizedNames

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Processing object type Collection.DM (6005)

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Opening connection to https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6005

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Will create object type at https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Opening connection to https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting icon for type: 6005 at url https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6005/ui/icon

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting localized names for type: 6005 at url https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6005/ui/localizedNames

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Error Message: Bad response from object type Collection.DM (6005) localized names creation - 500 at <br/><a href="https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6005/ui/localizedNames">https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6005/ui/localizedNames</a><br/>For troubleshooting tips, see SAS Note 66003 at<br/><a href="https://support.sas.com/kb/66003">https://support.sas.com/kb/66003</a>

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting output prop msg

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting output prop rc

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Processing object type CustomMetric.DM (6006)

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Opening connection to https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6006

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Encoded password

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Error Message: An error occurred retrieving information for object type CustomMetric.DM (6006) - 500 at <br/><a href="https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6006">https://ifrsdev.mbbank.com.vn:8343/SASWIPClientAccess/rest/types/objects/6006</a><br/>For troubleshooting tips, see SAS Note 66003 at<br/><a href="https://support.sas.com/kb/66003">https://support.sas.com/kb/66003</a>

[createObjectTypes.groovy]

[createObjectTypes.groovy]      [echo] Setting output prop msg

[gwootton]

Did you review the SAS Note referenced in that output (https://support.sas.com/kb/66003)?

A bind failure means we a port we are trying to use is already in use, but I'm not sure that is the current failure.

By default, the ports used by SASServer1_1 are 6969, 8080 or 8443 depending on TLS configuration (you can look at the port lines in catalina.properites), so I might check those using netstat or lsof to see if they are being used by another process if you are getting that message.

[alko13]

Hi,

Prior to configuration these ports are not in used. 
To add, I found out something weird. When I tried to manually stop SASServerX_X. It is complaining that the jmx port is already in use. But it is the same SASServerX_X is using these ports.

Example for SASServer1:

Other SASServerX_X are also complaining that their specified jmx port are in used when I tried to stop them.

Got idea why this is happening?

Thank you

[gwootton]

Sounds like the pid file that identifies the process to the control script (tcruntime-ctl.sh) is missing or wrong, so despite the process already running it thinks it is not and tries to start it again.

The file would be <SASConfig>/Levn/Web/WebAppServer/SASServerX_Y/logs/tcserver.pid, it should contain the PID for the server process.

[alko13]

Hi Greg, This is an unlikely behavior right? We have not done any special for this behavior to occur. Any suggestion to move forward? btw I also raise TS track 7613719762. But so far not much progress yet.

[gwootton]

The behavior suggests the control process was not used to stop the web application server (i.e. process killed versus stopped).
If you use tcruntime-ctl.sh to check the status of the server and it says it isn't running when it is this would confirm. To correct you would need to update the pid file with the correct process ID, or kill the existing server and start it using the tcruntime-ctl.sh command.
<SASConfig>/Levn/Web/WebAppServer/SASServerX_Y/bin/tcruntime-ctl.sh status
<SASConfig>/Levn/Web/WebAppServer/SASServerX_Y/bin/tcruntime-ctl.sh stop
<SASConfig>/Levn/Web/WebAppServer/SASServerX_Y/bin/tcruntime-ctl.sh start