Stored Process Error

[avinashginjupal]

Hello Balraj - Did you fixed the issue which was discussed here? we do have same problem with our STP and seeing same points discussed in thread, any help would be appreciated

Thanks

Avinash G

[Kurt_Bremser]

What exactly have you tried so far? Describe your tries, and post results. If you have already followed the suggestions I gave, post the relevant log parts.

[avinashginjupal]

We have verified whether Objectspawner is configured correctly or not - yes both Stored process server and Pooled Workspace Server are on the right hand side in Object Spawner Server Tab properties.

Under Content Server log and many other places I can see this error:

ERROR 2017-11-22 04:07:15,467 [RMI TCP Connection(158)-162.102.151.62] - failed to read bundle: f002de7c-b7f0-414d-8d8c-a6acee653dfa: java.io.EOFException
ERROR 2017-11-22 04:07:15,870 [RMI TCP Connection(158)-162.102.151.62] - failed to read bundle: 299fa6af-dbc2-444e-ae4b-90a47d17c5af: java.io.EOFException
ERROR 2017-11-22 04:07:16,434 [RMI TCP Connection(158)-162.102.151.62] - failed to read bundle: 423f6256-92e0-45ab-a455-c034c16c5405: java.io.EOFException

Under Stored Process Log :

2017-11-22 01:02:43,623 [JBoss Shutdown Hook] ERROR [] com.sas.svcs.authentication.helper.UserContextFactoryBean - An error occurred destroying user, sastrust@saspw. The Platform Services VM should be restarted.
2017-11-22 02:58:00,508 [http-0.0.0.0-8080-7] WARN [] com.sas.svcs.webapp.servlet.filters.SecurityFilter - The current HttpSession level entityKey does not match the value presented as a request parameter. The HttpSession will be invalidated and this request processed as a new HttpSession / User.

Ran STP Hello World with &_debug=2305

SASStoredProcess

>>> Stored Process Web Application Version 9.3 (Build 477) November 22, 2017 2:54:00 AM CST

>>> Using user name u431739 in domain DefaultAuth

>>> Parameter file /Params.config

>>> User context u431739

>>> Locale en_US

>>> Stored Process Input Parameters:

_debug = 2305
_grafloc = /sasweb/graph
_htcook = s_cc=true; s_sq=%5B%5BB%5D%5D; CSILL=1511252153462; JSESSIONID.SASServer1=64628F4BDF4C1624BE7AA0EBDE907FB8
_htua = Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
_program = /Products/SAS Intelligence Platform/Samples/Sample: Hello World
_reqmeth = GET
_rmtaddr = 10.92.227.201
_rmthost = 10.92.227.201
_srvname = crpra00a0021.wellsfargo.com
_srvport = 8080
_url = /SASStoredProcess/do
_userlocale = en_US
_username = u431739
_version = Version 9.3 (Build 477)
debug_action = form,properties,execute

>>> Servlet Session Data:

Servlet Session ID: 64628F4BDF4C1624BE7AA0EBDE907FB8
Counter: 16
Creation Time: Wed Nov 22 02:46:05 CST 2017
Last Accessed Time: Wed Nov 22 02:52:23 CST 2017 
Servlet Session Timeout: 30

>>> Using Pathurl SBIP://METASERVER/Products/SAS Intelligence Platform/Samples/Sample: Hello World(StoredProcess)

>>> Creating Stored Process Object

>>> Got Stored Process /Products/SAS Intelligence Platform/Samples/Sample: Hello World

Name: Sample: Hello World
Directory: /opt/sas/93tfcm/SASFoundation/9.3/samples/inttech
Source File: stphello.sas
Description: DATA Step-generated output using PUT statements.

>>> Setting stored process parameters

>>> Executing stored process object
Stored Process Error Unable to execute stored process.

Unable to connect to a SAS server.

com.sas.services.storedprocess.ExecutionException: A connection could not be obtained.
	at com.sas.services.storedprocess.Execution2.makeConnection(Execution2.java:337)
	at com.sas.services.storedprocess.Execution2.(Execution2.java:296)
	at com.sas.services.storedprocess.StoredProcess2.execute(StoredProcess2.java:1200)
	at com.sas.services.storedprocess.webapp.StoredProcessServlet.doRequest(StoredProcessServlet.java:1478)
	at com.sas.services.storedprocess.webapp.StoredProcessServlet.doGet(StoredProcessServlet.java:265)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.sas.svcs.webapp.servlet.filters.ThemeSupportFilter.onDoFilter(ThemeSupportFilter.java:177)
	at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.sas.svcs.webapp.servlet.filters.PlatformServicesFilter.onDoFilter(PlatformServicesFilter.java:148)
	at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.sas.svcs.webapp.servlet.filters.SecurityFilter.onDoFilter(SecurityFilter.java:212)
	at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.sas.services.storedprocess.webapp.SPFilter.onDoFilter(SPFilter.java:228)
	at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.sas.servlet.filters.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:62)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.sas.framework.webapp.servlet.SanitizingRequestFilter.onDoFilter(SanitizingRequestFilter.java:83)
	at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:66)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
	at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
	at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
	at java.lang.Thread.run(Thread.java:662)
Caused by: com.sas.services.connection.FatalConnectionFactoryException: The application could not log on to the server "crpra00a0021.wellsfargo.com:8601". No server is available at that port on that machine.
	at com.sas.services.connection.ClusterEnvelope.getConnection(ClusterEnvelope.java:235)
	at com.sas.services.connection.AggregationKernel.doGetConnection(AggregationKernel.java:241)
	at com.sas.services.connection.ConnectionFactoryKernel.getConnection(ConnectionFactoryKernel.java:325)
	at com.sas.services.connection.ConnectionFactoryShell.getConnection(ConnectionFactoryShell.java:75)
	at com.sas.services.connection.platform.PlatformConnectionFactory.getConnectionMetadata(PlatformConnectionFactory.java:571)
	at com.sas.services.connection.platform.PlatformConnectionFactory.getConnection(PlatformConnectionFactory.java:116)
	at com.sas.services.connection.platform.PlatformConnectionFactory.getConnection(PlatformConnectionFactory.java:94)
	at com.sas.services.storedprocess.ConnectionMaker.doMakeConnection(ConnectionMaker.java:117)
	at com.sas.services.storedprocess.ConnectionMaker.makeConnection(ConnectionMaker.java:61)
	at com.sas.services.storedprocess.Execution2.makeConnection(Execution2.java:330)
	... 57 more
Caused by: org.omg.CORBA.COMM_FAILURE: The application could not log on to the server "crpra00a0021.wellsfargo.com:8601". No server is available at that port on that machine.  vmcid: 0x0  minor code: 1  completed: No
	at com.sas.iom.orb.brg.Engine$5.run(Engine.java:3416)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.sas.iom.orb.brg.Engine.flowConnect(Engine.java:3490)
	at com.sas.iom.orb.brg.Engine.flow(Engine.java:658)
	at com.sas.iom.orb.brg.Engine.initClient(Engine.java:622)
	at com.sas.iom.orb.brg.ORBImpl.uri_to_object(ORBImpl.java:119)
	at com.sas.services.connection.ClusterEnvelope.createObject(ClusterEnvelope.java:278)
	at com.sas.services.connection.ClusterEnvelope.getConnection(ClusterEnvelope.java:73)
	... 66 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect(Native Method)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
	at com.sas.iom.orb.brg.Engine$5.run(Engine.java:3399)
	... 73 more

[JuanS_OCS]

Hello @avinashginjupal,

I would definitely check it the SAS metadata's sastrust@saspw or the OS' sassrv account are ok on the password, or just locked out.

Also, I think perhaps it would be an idea to involve SAS Technical Support, since they can see/touch your system safely.

[avinashginjupal]

I feel passwords are not causing the issue here, something in the background that makes connections to required services is not working properly.

I feel the ports we are trying to call are not listening to application

[Kurt_Bremser]

Take a look at the Object Spawner logs

(...../lev1/ObjectSpawner/Logs)

The last file name there will contain the process number of the spawner currently running. Look for the first file (datewise) that contains that process number. When you open that, you can see the initialization of the spawner, and you'll find entries for every port it listens to. 8601 should be found there. If not, there's a misconfiguration or somehow sastrust@saspw could not read the spawner's configuration, or a server configuration that it should have access to.

If, OTOH, the spawner set up listening correctly, then you have a connection problem on the net, not in SAS.

[Kurt_Bremser]

To me that looks as if your MidTier (the web application server) cannot connect to crpra00a0021.wellsfargo.com:8601

This might be because a firewall interferes with the connection. Have your network people check that.

[avinashginjupal]

This is standalone server, Metadata, Compute and Mid runs on same server.

[Kurt_Bremser]

Then you need to check if the Object Spawner is up and running, and check it's logs as I already suggested.

[avinashginjupal]

Object Spawner is up and running, I doubt Content Server as it is producing below errors:

ERROR 2017-11-22 02:52:02,878 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: 3e00bcdb-1b5e-4f00-b21a-b83e45ed8011: java.io.EOFException
ERROR 2017-11-22 02:52:03,402 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: f002de7c-b7f0-414d-8d8c-a6acee653dfa: java.io.EOFException
ERROR 2017-11-22 02:52:03,802 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: 299fa6af-dbc2-444e-ae4b-90a47d17c5af: java.io.EOFException
ERROR 2017-11-22 02:52:04,362 [RMI TCP Connection(88)-162.102.151.62] - failed to read bundle: 423f6256-92e0-45ab-a455-c034c16c5405: java.io.EOFException

Even today I see only these 4 repeatedly, I think I should fixed these by going to content server repository. What do you say? 

ERROR 2017-11-23 00:37:02,828 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: 3e00bcdb-1b5e-4f00-b21a-b83e45ed8011: java.io.EOFException
ERROR 2017-11-23 00:37:03,160 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: f002de7c-b7f0-414d-8d8c-a6acee653dfa: java.io.EOFException
ERROR 2017-11-23 00:37:03,497 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: 299fa6af-dbc2-444e-ae4b-90a47d17c5af: java.io.EOFException
ERROR 2017-11-23 00:37:03,970 [RMI TCP Connection(1361)-162.102.151.62] - failed to read bundle: 423f6256-92e0-45ab-a455-c034c16c5405: java.io.EOFException

[Kurt_Bremser]

Have you already opened a track with SAS technical support? If not, do so now.

[avinashginjupal]

yes there is track running with SAS on this along with our other issue and meanwhile I have found that port 8601 is not listening on server, raised ticket to our UNIX team to check that as I can see 8601 port is listening in all other servers in other environments.

[avinashginjupal]

Issue is fixed

Resolution:

Removed SAS general servers group from Metadata server: unrestricted Role

Further checking the logs after restart found access issues with starting pooler workspace and stored process server.

so based on error found that “ SAS General servers group was part of Metadata servers: Unrestricted Properties”  so removed this group and restarted Object spawner. It fixed the issue.

2017-11-23T06:05:39,216 INFO  [00000024] sas1 - Reserved IPv4 port 8701 for server listen (connection 4).

2017-11-23T06:05:39,284 ERROR [00000024] sas1 - Access denied.

2017-11-23T06:05:39,284 INFO  [00000024] sas1 - Access denied.

2017-11-23T06:05:39,284 WARN  [00000024] sas1 - The credentials specified for the SASApp - Pooled Workspace Server (A54KOR8M.AV000007) server definition failed to authenticate.  Therefore this server definition will not be included.

2017-11-23T06:05:39,288 WARN  [00000024] sas1 - The password obtained from the metadata indicates that it was obtained by an user that has unrestricted access.  Please check your configuration.

2017-11-23T06:05:39,288 INFO  [00000024] sas1 - Reserved IPv4 port 8601 for server listen (connection 5).