Portal/Stored Process Server "freezes"

[PhilMason]

We are using the Portal at my client (a Pharmaceutical company). They are running stored processes which display patient profiles in the portal. After using the system for a while (between 30 mins and several hours) it will sit there apparently running the stored process with the progress bar creeping slowly along for several minutes until it finally just finishes and shows nothing. Sometimes we can get an error such as "Stored Process Error".

If I go to Management Console and "Test Connection" on the logical stored process server then I get "sam.S249.ex.msg: A connection to server host x.x.x.x could not be established on port xxxx. Either there is no server listening for connections or the server is too busy to accept any new connections. Make sure the server has started and that the values provided for host and port are correct."

After this no one will be able to run any stored processes in the system. We have taken to restarting the Object Spawner at this point which restores everything to normal operation. But after some more use of the system we get back to this point again.
Our enthusiastic users are getting more and more annoyed. I fear that if we cant make the portal more reliable very soon then they will demand an alternative system.
We only have about 5 users of this portal currently.

1 - Has anyone seen this kind of problem?

2 - Does anyone have a solution?

thanks
Phil

[deleted_user]

Although I think this is an issue that you must track with your local Technical Support office, this SAS-Note might be of help:

http://support.sas.com/techsup/unotes/SN/020/020387.html

Cheers.

[PhilMason]

thanks - I did see that usage note and made the changes to the load balancing algorithm - but it didnt fix the problem. We also raised it with tech support and sent them logs, but they havent got back to us yet. I wanted to raise it here to see if others had come across it. Seems like they probably havent (which is good) so it may be something more specific to us.

[MarkBodt_NZ]

Hi Phil,

I'm not sure if the problem we had a couple of months ago is similar to yours. We've got portlets displaying actual charts and tabular reports. These are built on the fly using stored processes. i.e. when the portal user diaplays a portal page, the charts display without them having to make any selections or click anything.

When we had more than one portlet displaying STP output i.e. when the page was displayed, several STPs kicked off processing, we found that intermittently, one or the other STPs would fail and would either display nothing, display "Stored Process Error" or "Session context destoyed" or "Error while getting the root..." .

Which one would fail seemed to be random and refreshing the browser would sometimes result in different a STP portlet failing and one that failed before now working. Often they would all correctly display output after a page refresh. I found that more portlets displaying STP output on a page, the higher the likelihood that one or more would fail.

At the time we logged a track: SAS au1037195 and they sent us a new sas.svc.storedprocess.webapp.jar which was at the time a dev version. It was supposedly to be released as a hotfix in August. This fixed the problem for us.

Like I say, This may not be the same as what you're experiencing, but it might be of help.

Mark

[PhilMason]

Our problem has been escalated to SAS in Cary. We are following various leads and have even installed an experimental fix - which seemed to help but not completely solving our problem.

I did come up with a way to kind of "fix" the problem. The main issue is that our users were suffering outages and could not work when the stored process servers would hang. They would have to contact us, we had to restart the spawner and kill orphan processes. However I increased the number of servers from 6 up to 31 in management console!!! Rather than having to restart several times a day, it has currently been running for 6 days and we have run over 4,000 stored processes. We are currently using 27 stored process servers - most of which are probably hung and swapped out. As far as the users are concerned they think the problem is fixed - although we have merely extending the time between having to restart the spawner.