Configure Middle Tier Error - Application Server has started, however web applications failed

[MariaD]

Hi folks, 

We're installing a web node (middle tier) on Oracle Linux. We have SAS 9.4 M6 version but on step 6 o 7 (Starting SAS Web Application Server Node Configuration) the following error occurs:

Application Server has started, however web application with the following context failed to start...

On the configuration logs appears the following message:

[appserverTarget] 2020-01-07 17:23:04,948 [main] INFO com.sas.appserver.utils.AntProcessInvoker - Arguments:
[appserverTarget] 2020-01-07 17:23:04,948 [main] INFO com.sas.appserver.utils.AntProcessInvoker - start
[appserverTarget] 2020-01-07 17:23:04,949 [main] INFO com.sas.appserver.utils.AntProcessInvoker - spawning process
[appserverTarget] 2020-01-07 17:23:04,951 [main] INFO com.sas.appserver.vfabrictcsvr.Server - XX.XX.XX.XX SASServer12 using http on port 9180
[appserverTarget] 2020-01-07 17:23:14,951 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable(XX.XX.XX.XX,9180) >>>
[appserverTarget] 2020-01-07 17:23:14,952 [main] INFO com.sas.appserver.utils.PortUtils - Connection refused (Connection refused)
[appserverTarget] 2020-01-07 17:23:14,952 [main] INFO com.sas.appserver.utils.PortUtils - isAvailable=true
[appserverTarget] 2020-01-07 17:23:14,952 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable() <<<
[appserverTarget] 2020-01-07 17:23:14,953 [main] INFO com.sas.appserver.vfabrictcsvr.Server - Waiting for port (9180) to open on address (10.192.29.37).
[appserverTarget] 2020-01-07 17:23:24,953 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable(XX.XX.XX.XX,9180) >>>
[appserverTarget] 2020-01-07 17:23:24,954 [main] INFO com.sas.appserver.utils.PortUtils - Connection refused (Connection refused)
[appserverTarget] 2020-01-07 17:23:24,954 [main] INFO com.sas.appserver.utils.PortUtils - isAvailable=true
[appserverTarget] 2020-01-07 17:23:24,954 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable() <<<
[appserverTarget] 2020-01-07 17:23:24,954 [main] INFO com.sas.appserver.vfabrictcsvr.Server - Waiting for port (9180) to open on address (10.192.29.37).
[appserverTarget] 2020-01-07 17:23:34,955 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable(XX.XX.XX.XX,9180) >>>
[appserverTarget] 2020-01-07 17:23:34,955 [main] INFO com.sas.appserver.utils.PortUtils - Connection refused (Connection refused)
[appserverTarget] 2020-01-07 17:23:34,956 [main] INFO com.sas.appserver.utils.PortUtils - isAvailable=true
[appserverTarget] 2020-01-07 17:23:34,956 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable() <<<
[appserverTarget] 2020-01-07 17:23:34,958 [main] INFO com.sas.appserver.vfabrictcsvr.Server - Waiting for port (9180) to open on address (10.192.29.37).
[appserverTarget] 2020-01-07 17:23:44,958 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable(XX.XX.XX.XX,9180) >>>
[appserverTarget] 2020-01-07 17:23:44,960 [main] INFO com.sas.appserver.utils.PortUtils - Connection refused (Connection refused)
[appserverTarget] 2020-01-07 17:23:44,960 [main] INFO com.sas.appserver.utils.PortUtils - isAvailable=true
[appserverTarget] 2020-01-07 17:23:44,960 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable() <<<
[appserverTarget] 2020-01-07 17:23:44,965 [main] INFO com.sas.appserver.vfabrictcsvr.Server - Waiting for port (9180) to open on address (10.192.29.37).
[appserverTarget] 2020-01-07 17:23:54,967 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable(XX.XX.XX.XX,9180) >>>
[appserverTarget] 2020-01-07 17:23:54,967 [main] INFO com.sas.appserver.utils.PortUtils - isAvailable=false
[appserverTarget] 2020-01-07 17:23:54,967 [main] INFO com.sas.appserver.utils.PortUtils - isPortAvailable() <<<
[appserverTarget] 2020-01-07 17:23:54,978 [main] INFO com.sas.appserver.vfabrictcsvr.Server - Server SASServer12_1 started successfully.
[appserverTarget] 2020-01-07 17:23:54,979 [main] INFO com.sas.appserver.vfabrictcsvr.Server - testServer() >>>
[appserverTarget] 2020-01-07 17:23:54,979 [main] INFO com.sas.appserver.vfabrictcsvr.Server - getFailedApps() >>>
[appserverTarget] 2020-01-07 17:23:59,983 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASLASRAuthorization context failed to start
[appserverTarget] 2020-01-07 17:23:59,984 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASSearchService context failed to start
[appserverTarget] 2020-01-07 17:23:59,986 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalyticsHub context failed to start
[appserverTarget] 2020-01-07 17:23:59,988 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalytics context failed to start
[appserverTarget] 2020-01-07 17:23:59,989 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalyticsExplorer context failed to start
[appserverTarget] 2020-01-07 17:23:59,990 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalyticsGraphBuilder context failed to start
[appserverTarget] 2020-01-07 17:23:59,991 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualDataBuilder context failed to start
[appserverTarget] 2020-01-07 17:23:59,992 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalyticsDesigner context failed to start
[appserverTarget] 2020-01-07 17:23:59,993 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalyticsTransport context failed to start
[appserverTarget] 2020-01-07 17:23:59,994 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalyticsViewer context failed to start
[appserverTarget] 2020-01-07 17:23:59,994 [main] ERROR com.sas.appserver.vfabrictcsvr.Server - Webapp with SASVisualAnalyticsAdministrator context failed to start

We tried to continue de installation and the restart the services. The services started ok (showing all services UP) but when you login on SAS VA Administrator an error occurs. View the log in Lev1/Web/Logs/SASServer12_1, we find the foliowing error:

020-01-07 17:23:49,167 [localhost-startStop-8] ERROR [unknownunknown] org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.sas.svcs.springframework.config.DefaultProvidedPropertyPlaceholderConfigurer#0': Cannot resolve reference to bean 'environmentProperties' while setting bean property 'properties'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environmentProperties': FactoryBean threw exception on object creation; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.atomikos.jdbc.AtomikosSQLException: Failed to grow the connection pool

We executed the same procedure on our development environment and everything works fine. Any idea?

Regards,

[rbetancourt]

My experiences with mid-tier deployments is you are not likely to determine the root cause even if you have uber-skills in forming Google search queries.  You should contact Technical Support.

There are numerous points of integration throughout this entire stack.  And sometimes, the mid-tier administration guide has incorrect/imcomplete instructions.  Last fall I reported a documentation defect for horizontally scaling the mid-tier's JMS server.  The XML they documented was incomplete.

The connection refused error can point to any number of issues.  One possibility is when you re-started the services, there was an existing JVM process that was already listening on the port assigned to the process.

I'm assuming you are on Linux, have installed the netstat package and have root access.  You can verify what processes are listening on each port with:

# netstat -antp | grep "LISTEN"

 Before restarting any service, I use the above command to find the PID associated with the process that "listens" on a given port.  If a script existing to stop this process, Icall it to stop the current process.  Otherwise:

# kill -9 <PID>

 Best of luck!

[AnandVyas]

Hi @MariaD 

Can you check on your compute node if services are up or not? Below error is pointing out to that:

Could not get JDBC Connection; nested exception is com.atomikos.jdbc.AtomikosSQLException: Failed to grow the connection pool

Usually this could be anyone of the internal SAS database like WIPDS..

[MariaD]

Hi @AnandVyas ,

Yes, the service is up on the compute node. We already have a one web node (middle tier) and works fine. We have problems configuring the second middle tier node. 

Regards,