Web Application Server Configuration error

AsakoOkano — Wed, 31 Aug 2016 07:50:41 GMT

I am trying to configure SAS9.4 and Visual Analytics Server on two Linux Machine.

Machine1:Metadata Server and SAS AppServer using Mid-Tier on Machine2

Machine2:Mid-Tier and Visual Analytics App Server, using Metadata Server on Machine1

On Machine1 SAS Deployment Manager configuration completed successfully and Metadata server runs properly.

But on Machine2 SAS Deployment Manager fail to configure Web Application Server.

It seems cache locater on Machine1 and Machne2 have some problems with their connection,and Web App ended up failing to estabish port 8080.

Both of Machines don't run iptables and SELinux. To connect Metadata from Machine2 to Machine1 is fine.

Can you please help what is wrong with this?

↓vfabrictcsvr_startMidtierServers_2016-08-30-21.09.55.log

---------------------------------------------

[root@test01 Configure]# cat vfabrictcsvr_startMidtierServers_2016-08-30-21.09.55.log | grep ERROR
[appserverTarget] [exec] ERROR: Operation "stop-locator" failed because: Locator in directory "/opt/03_SAS/config/Lev1/Web/gemfire/instances/ins_41415" is not running..
2016-08-30 22:01:52,471 [main] ERROR com.sas.sdw.SDWExceptionHandler - com.sas.appserver.utils.AppServerException: Timeout for Application Server startup has reached before port opened.

------

[appserverTarget] 2016-08-30 22:01:39,035 [main] INFO com.sas.appserver.vfabrictcsvr.Server - Waiting for port (8080) to open on address (172.17.134.73).
[appserverTarget] 2016-08-30 22:01:39,035 [main] INFO com.sas.appserver.vfabrictcsvr.Server - Timeout reached before port opened. (172.17.134.73:8080)
[appserverTarget] 2016-08-30 22:01:39,127 [main] INFO com.sas.appserver.vfabrictcsvr.Server - getFailedApps() >>>
[appserverTarget] 2016-08-30 22:01:47,898 [main] INFO com.sas.appserver.vfabrictcsvr.Server - getFailedApps() >>>
[appserverTarget] 2016-08-30 22:01:47,898 [main] INFO com.sas.appserver.vfabrictcsvr.Server - testServer() - Throwing AppServerException
[appserverTarget] 2016-08-30 22:01:47,898 [main] INFO com.sas.appserver.vfabrictcsvr.Server - testServer() <<<
[appserverTarget] 2016-08-30 22:01:50,205 [main] INFO com.sas.sdw.SDWExceptionHandler - handle() >>>
[appserverTarget] 2016-08-30 22:01:50,206 [main] INFO com.sas.sdw.SDWExceptionHandler - handleSDWException() >>>
[appserverTarget] 2016-08-30 22:01:50,501 [main] INFO com.sas.sdw.SDWExceptionHandler - target=startMidtierServers, task=com.sas.sdw.SDWExceptionHandler
[appserverTarget] 2016-08-30 22:01:50,501 [main] INFO com.sas.sdw.SDWExceptionHandler - message=Timeout for Application Server startup has reached before port opened.
[propertyfile] Creating new property file: /opt/03_SAS/config/Lev1/Logs/Configure/vfabrictcsvr_config_status.properties
[propertyfile] Updating property file: /opt/03_SAS/config/Lev1/Logs/Configure/vfabrictcsvr_config_status.properties
2016-08-30 22:01:51,102 [main] INFO com.sas.sdw.SDWExceptionHandler - configStatusFile set.
2016-08-30 22:01:52,471 [main] ERROR com.sas.sdw.SDWExceptionHandler - com.sas.appserver.utils.AppServerException: Timeout for Application Server startup has reached before port opened.

---------------------------------------------

↓server.log

---------------------------------------------

[root@test01 logs]# cat server.log | egrep "WARN|ERROR"
2016-08-30 21:10:26,625 WARN (main) [com.springsource.tcserver.serviceability.rmi.JmxSocketListener] Detected LDAP configuration '/opt/03_SAS/config/Lev1/Web/WebAppServer/SASServer1_1/conf/jaas.config', but there is no ldap configuration entry specified.
2016-08-31 01:53:19,358 WARN (schedulingListenerContainer-1) [org.apache.activemq.TransactionContext] prepare of: XID:[1096044365,globalId=61746f6d696b6f733136303532303831343431383433353030303033,branchId=61746f6d696b6f73313630353230383134343735363237] failed with: javax.jms.JMSException: java.io.EOFException
2016-08-31 04:38:48,169 WARN (workflowEventListenerContainer-3) [org.apache.activemq.TransactionContext] prepare of: XID:[1096044365,globalId=61746f6d696b6f733136303532303831343431323633363030303034,branchId=61746f6d696b6f73313630353230383134343938383232] failed with: javax.jms.JMSException: Cannot send, channel has already failed: tcp://172.17.134.73:61616
2016-08-31 07:07:42,488 WARN (workflowEventListenerContainer-5) [org.apache.activemq.TransactionContext] prepare of: XID:[1096044365,globalId=61746f6d696b6f733136303532303831343433303231373030303034,branchId=61746f6d696b6f7331363035323038313434313133363831] failed with: javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp://172.17.134.73:61616
2016-08-31 09:54:28,521 WARN (policyMessageListenerContainer-41) [org.apache.activemq.TransactionContext] prepare of: XID:[1096044365,globalId=61746f6d696b6f733136303532303831343431373337363030303035,branchId=61746f6d696b6f7331363035323038313434313239383830] failed with: javax.jms.JMSException: Cannot send, channel has already failed: tcp://172.17.134.73:61616
2016-08-31 09:54:31,672 WARN (policyMessageListenerContainer-40) [org.apache.activemq.TransactionContext] prepare of: XID:[1096044365,globalId=61746f6d696b6f733136303532303831343431373337353030303035,branchId=61746f6d696b6f7331363035323038313434313239383739] failed with: javax.jms.JMSException: Cannot send, channel has already failed: tcp://172.17.134.73:61616

↓gemfire.log

---------------------------------------------

[info 2016/08/30 21:10:13.695 JST <main> tid=0x1] Starting peer location for Distribution Locator on MACHINE2[41415]
[info 2016/08/30 21:10:13.710 JST <main> tid=0x1] Starting Distribution Locator on MACHINE2[41415]
[info 2016/08/30 21:10:13.818 JST <main> tid=0x1] MACHINE2 [ 41,415 ] attempting to get state from MACHINE1.MYDOMAIN [ 41,415 ]
[info 2016/08/30 21:10:13.948 JST <main> tid=0x1] Initial discovery set is [mbr=MACHINE1(1672)<v0>:26096]
[config 2016/08/30 21:10:14.951 JST <main> tid=0x1] multicast is disabled
[info 2016/08/30 21:10:14.952 JST <main> tid=0x1] Starting distributed system
[info 2016/08/30 21:10:15.239 JST <main> tid=0x1] GemFire P2P Listener started on tcp:///172.17.134.73:46368
[info 2016/08/30 21:10:16.164 JST <main> tid=0x1] Attempting to join distributed system whose membership coordinator is MACHINE1(1672)<v0>:26096 using membership ID MACHINE2(31939):60924
[info 2016/08/30 21:10:19.899 JST <main> tid=0x1] Entered into membership in group GF66 with ID MACHINE2(31939:admin)<v1>:60924/46368.
[info 2016/08/30 21:10:19.900 JST <main> tid=0x1] Starting DistributionManager MACHINE2(31939:admin)<v1>:60924/46368.
[info 2016/08/30 21:10:19.902 JST <main> tid=0x1] Initial (membershipManager) view = [MACHINE1(1672:admin)<v0>:26096/49985, MACHINE2(31939:admin)<v1>:60924/46368]
[info 2016/08/30 21:10:19.903 JST <main> tid=0x1] DMMembership: Admitting new administration member < MACHINE1(1672:admin)<v0>:26096/49985 >.
[info 2016/08/30 21:10:19.903 JST <main> tid=0x1] DMMembership: Admitting new administration member < MACHINE2(31939:admin)<v1>:60924/46368 >.
[info 2016/08/30 21:10:20.131 JST <main> tid=0x1] DistributionManager MACHINE2(31939:admin)<v1>:60924/46368 started on MACHINE2.MYDOMAIN[41415],MACHINE1.MYDOMAIN[41415]. There were 0 other DMs. others: [] (admin only)
[info 2016/08/30 21:10:20.152 JST <main> tid=0x1] Locator started on 172.17.134.73[41415]
[info 2016/08/30 21:10:20.152 JST <main> tid=0x1] Starting server location for Distribution Locator on MACHINE2[41415]
[info 2016/08/30 21:10:20.253 JST <Thread-7 StatSampler> tid=0x28] Disabling statistic archival.
[info 2016/08/30 21:10:54.525 JST <P2P message reader@3d9f4563> tid=0x2a] Admitting member <MACHINE2(32056)<v2>:39247/50976>. Now there are 3 non-admin member(s).

Asako

Re: Web Application Server Configuration error

JuanS_OCS — Wed, 31 Aug 2016 08:00:40 GMT

Hi, your Cache Locator seems to be good, but you have got problems with the JMS service.

For testing ports you can always use telnet (if available on your server) or other tools to test ports and connectivity.

Also, you can use netstat -nao to search for currently open ports.

Besides this, I would check if there is any other firewall besides the iptables, between both servers, and I would check if you have multiple hostnames/ips on the 2 servers.

Re: Web Application Server Configuration error

JasonDiVirgilio — Thu, 16 Nov 2017 14:36:22 GMT

"...and I would check if you have multiple hostnames/ips on the 2 servers."

This comment from JuanS_OCS just saved me big time on an installation. An incorrect IP address in our /etc/hosts file on the middle tier was causing the error mentioned at the top (and probably all of my "port #NNNN unavailable" errors ), causing a lot of troubleshooting and headaches.

Re: Web Application Server Configuration error

JuanS_OCS — Thu, 16 Nov 2017 14:56:34 GMT

Hello @JasonDiVirgilio,

I am really glad some of it pin-pointed you to the right direction, and nice catch 🙂

Re: Web Application Server Configuration error

AsakoOkano — Fri, 17 Nov 2017 02:00:45 GMT

I am sorry that I had not closed this QA. As you mentioned, disabling iptables and SELinux worked well for my environment.
Thank you for your advise!

Re: Web Application Server Configuration error

JuanS_OCS — Fri, 17 Nov 2017 08:13:30 GMT

Nice and well done.

Glad to read it works now.

topic Re: Web Application Server Configuration error in Administration and Deployment

Web Application Server Configuration error

Re: Web Application Server Configuration error

Re: Web Application Server Configuration error

Re: Web Application Server Configuration error

Re: Web Application Server Configuration error

Re: Web Application Server Configuration error