Hello Balraj - Did you fixed the issue which was discussed here? we do have same problem with our STP and seeing same points discussed in thread, any help would be appreciated
Thanks
Avinash G
What exactly have you tried so far? Describe your tries, and post results. If you have already followed the suggestions I gave, post the relevant log parts.
We have verified whether Objectspawner is configured correctly or not - yes both Stored process server and Pooled Workspace Server are on the right hand side in Object Spawner Server Tab properties.
Under Content Server log and many other places I can see this error:
ERROR 2017-11-22 04:07:15,467 [RMI TCP Connection(158)-162.102.151.62] - failed to read bundle: f002de7c-b7f0-414d-8d8c-a6acee653dfa: java.io.EOFException
ERROR 2017-11-22 04:07:15,870 [RMI TCP Connection(158)-162.102.151.62] - failed to read bundle: 299fa6af-dbc2-444e-ae4b-90a47d17c5af: java.io.EOFException
ERROR 2017-11-22 04:07:16,434 [RMI TCP Connection(158)-162.102.151.62] - failed to read bundle: 423f6256-92e0-45ab-a455-c034c16c5405: java.io.EOFException
Under Stored Process Log :
2017-11-22 01:02:43,623 [JBoss Shutdown Hook] ERROR [] com.sas.svcs.authentication.helper.UserContextFactoryBean - An error occurred destroying user, sastrust@saspw. The Platform Services VM should be restarted.
2017-11-22 02:58:00,508 [http-0.0.0.0-8080-7] WARN [] com.sas.svcs.webapp.servlet.filters.SecurityFilter - The current HttpSession level entityKey does not match the value presented as a request parameter. The HttpSession will be invalidated and this request processed as a new HttpSession / User.
Ran STP Hello World with &_debug=2305
SASStoredProcess
_debug = 2305 _grafloc = /sasweb/graph _htcook = s_cc=true; s_sq=%5B%5BB%5D%5D; CSILL=1511252153462; JSESSIONID.SASServer1=64628F4BDF4C1624BE7AA0EBDE907FB8 _htua = Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko _program = /Products/SAS Intelligence Platform/Samples/Sample: Hello World _reqmeth = GET _rmtaddr = 10.92.227.201 _rmthost = 10.92.227.201 _srvname = crpra00a0021.wellsfargo.com _srvport = 8080 _url = /SASStoredProcess/do _userlocale = en_US _username = u431739 _version = Version 9.3 (Build 477) debug_action = form,properties,execute
Servlet Session ID: 64628F4BDF4C1624BE7AA0EBDE907FB8 Counter: 16 Creation Time: Wed Nov 22 02:46:05 CST 2017 Last Accessed Time: Wed Nov 22 02:52:23 CST 2017 Servlet Session Timeout: 30
Name: Sample: Hello World Directory: /opt/sas/93tfcm/SASFoundation/9.3/samples/inttech Source File: stphello.sas Description: DATA Step-generated output using PUT statements.
Unable to connect to a SAS server.
com.sas.services.storedprocess.ExecutionException: A connection could not be obtained. at com.sas.services.storedprocess.Execution2.makeConnection(Execution2.java:337) at com.sas.services.storedprocess.Execution2.(Execution2.java:296) at com.sas.services.storedprocess.StoredProcess2.execute(StoredProcess2.java:1200) at com.sas.services.storedprocess.webapp.StoredProcessServlet.doRequest(StoredProcessServlet.java:1478) at com.sas.services.storedprocess.webapp.StoredProcessServlet.doGet(StoredProcessServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.sas.svcs.webapp.servlet.filters.ThemeSupportFilter.onDoFilter(ThemeSupportFilter.java:177) at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.sas.svcs.webapp.servlet.filters.PlatformServicesFilter.onDoFilter(PlatformServicesFilter.java:148) at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.sas.svcs.webapp.servlet.filters.SecurityFilter.onDoFilter(SecurityFilter.java:212) at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.sas.services.storedprocess.webapp.SPFilter.onDoFilter(SPFilter.java:228) at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.sas.servlet.filters.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:62) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.sas.framework.webapp.servlet.SanitizingRequestFilter.onDoFilter(SanitizingRequestFilter.java:83) at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:662) Caused by: com.sas.services.connection.FatalConnectionFactoryException: The application could not log on to the server "crpra00a0021.wellsfargo.com:8601". No server is available at that port on that machine. at com.sas.services.connection.ClusterEnvelope.getConnection(ClusterEnvelope.java:235) at com.sas.services.connection.AggregationKernel.doGetConnection(AggregationKernel.java:241) at com.sas.services.connection.ConnectionFactoryKernel.getConnection(ConnectionFactoryKernel.java:325) at com.sas.services.connection.ConnectionFactoryShell.getConnection(ConnectionFactoryShell.java:75) at com.sas.services.connection.platform.PlatformConnectionFactory.getConnectionMetadata(PlatformConnectionFactory.java:571) at com.sas.services.connection.platform.PlatformConnectionFactory.getConnection(PlatformConnectionFactory.java:116) at com.sas.services.connection.platform.PlatformConnectionFactory.getConnection(PlatformConnectionFactory.java:94) at com.sas.services.storedprocess.ConnectionMaker.doMakeConnection(ConnectionMaker.java:117) at com.sas.services.storedprocess.ConnectionMaker.makeConnection(ConnectionMaker.java:61) at com.sas.services.storedprocess.Execution2.makeConnection(Execution2.java:330) ... 57 more Caused by: org.omg.CORBA.COMM_FAILURE: The application could not log on to the server "crpra00a0021.wellsfargo.com:8601". No server is available at that port on that machine. vmcid: 0x0 minor code: 1 completed: No at com.sas.iom.orb.brg.Engine$5.run(Engine.java:3416) at java.security.AccessController.doPrivileged(Native Method) at com.sas.iom.orb.brg.Engine.flowConnect(Engine.java:3490) at com.sas.iom.orb.brg.Engine.flow(Engine.java:658) at com.sas.iom.orb.brg.Engine.initClient(Engine.java:622) at com.sas.iom.orb.brg.ORBImpl.uri_to_object(ORBImpl.java:119) at com.sas.services.connection.ClusterEnvelope.createObject(ClusterEnvelope.java:278) at com.sas.services.connection.ClusterEnvelope.getConnection(ClusterEnvelope.java:73) ... 66 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500) at com.sas.iom.orb.brg.Engine$5.run(Engine.java:3399) ... 73 more
Hello @avinashginjupal,
I would definitely check it the SAS metadata's sastrust@saspw or the OS' sassrv account are ok on the password, or just locked out.
Also, I think perhaps it would be an idea to involve SAS Technical Support, since they can see/touch your system safely.
I feel passwords are not causing the issue here, something in the background that makes connections to required services is not working properly.
I feel the ports we are trying to call are not listening to application
Take a look at the Object Spawner logs
(...../lev1/ObjectSpawner/Logs)
The last file name there will contain the process number of the spawner currently running. Look for the first file (datewise) that contains that process number. When you open that, you can see the initialization of the spawner, and you'll find entries for every port it listens to. 8601 should be found there. If not, there's a misconfiguration or somehow sastrust@saspw could not read the spawner's configuration, or a server configuration that it should have access to.
If, OTOH, the spawner set up listening correctly, then you have a connection problem on the net, not in SAS.
To me that looks as if your MidTier (the web application server) cannot connect to crpra00a0021.wellsfargo.com:8601
This might be because a firewall interferes with the connection. Have your network people check that.
This is standalone server, Metadata, Compute and Mid runs on same server.
Then you need to check if the Object Spawner is up and running, and check it's logs as I already suggested.
Object Spawner is up and running, I doubt Content Server as it is producing below errors:
ERROR 2017-11-22 02:52:02,878 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: 3e00bcdb-1b5e-4f00-b21a-b83e45ed8011: java.io.EOFException
ERROR 2017-11-22 02:52:03,402 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: f002de7c-b7f0-414d-8d8c-a6acee653dfa: java.io.EOFException
ERROR 2017-11-22 02:52:03,802 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: 299fa6af-dbc2-444e-ae4b-90a47d17c5af: java.io.EOFException
ERROR 2017-11-22 02:52:04,362 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: 423f6256-92e0-45ab-a455-c034c16c5405: java.io.EOFException
Even today I see only these 4 repeatedly, I think I should fixed these by going to content server repository. What do you say?
ERROR 2017-11-23 00:37:02,828 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: 3e00bcdb-1b5e-4f00-b21a-b83e45ed8011: java.io.EOFException
ERROR 2017-11-23 00:37:03,160 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: f002de7c-b7f0-414d-8d8c-a6acee653dfa: java.io.EOFException
ERROR 2017-11-23 00:37:03,497 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: 299fa6af-dbc2-444e-ae4b-90a47d17c5af: java.io.EOFException
ERROR 2017-11-23 00:37:03,970 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: 423f6256-92e0-45ab-a455-c034c16c5405: java.io.EOFException
Have you already opened a track with SAS technical support? If not, do so now.
yes there is track running with SAS on this along with our other issue and meanwhile I have found that port 8601 is not listening on server, raised ticket to our UNIX team to check that as I can see 8601 port is listening in all other servers in other environments.
Issue is fixed
Resolution:
Removed SAS general servers group from Metadata server: unrestricted Role
Further checking the logs after restart found access issues with starting pooler workspace and stored process server.
so based on error found that “ SAS General servers group was part of Metadata servers: Unrestricted Properties” so removed this group and restarted Object spawner. It fixed the issue.
2017-11-23T06:05:39,216 INFO [00000024] sas1 - Reserved IPv4 port 8701 for server listen (connection 4).
2017-11-23T06:05:39,284 ERROR [00000024] sas1 - Access denied.
2017-11-23T06:05:39,284 INFO [00000024] sas1 - Access denied.
2017-11-23T06:05:39,284 WARN [00000024] sas1 - The credentials specified for the SASApp - Pooled Workspace Server (A54KOR8M.AV000007) server definition failed to authenticate. Therefore this server definition will not be included.
2017-11-23T06:05:39,288 WARN [00000024] sas1 - The password obtained from the metadata indicates that it was obtained by an user that has unrestricted access. Please check your configuration.
2017-11-23T06:05:39,288 INFO [00000024] sas1 - Reserved IPv4 port 8601 for server listen (connection 5).
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
See how to use one filter for multiple data sources by mapping your data from SAS’ Alexandria McCall.
Find more tutorials on the SAS Users YouTube channel.