06-24-2016 07:57 AM
We have been using SAS9.4M2 BI Platform for about 2 years now, and have had endless problems with our Middle tier.
SAS VA or SPS would work for a few days and then drop offline, running out of PostgrssSQL connections etc.
Only recently, after paying for Consultation with SAS we learned how troublesome the Web Services and JVMs on SAS9.4M2 are, with some unpublished defects with no hotfixes.
Last week, after bringing some stability to our Web Tier, (SPS Server, Environment Manager, SAS VA, etc) we were able to do some work for a week, when suddenly this week, nobody in our SAS User community is able to log in as they get an error message saying they have Invalid Usernames and Passwords.
SAS is telling us we need to upgrade to SAS9.4.M3 asap, but that won't be happening in the immediate future. So we are stuck with Mid Tier defects until then.
06-24-2016 08:19 AM
thank you for bringing your question to the SAS Communities.
Without having more knowledge of your problems and your SAS deployment and configuration, my best suggestion is that better you keep aligned with SAS, since only they can know what can be better for your platform health.
Anyway, what you can do is try to understand better the configurations on the middle tier related to the connections you just mentioned:
For this you have 2 great resources: SAS(R) 9.4 Intelligence Platform: Middle-Tier Administration Guide, Third Edition and SAS(R) 9.4 Web Applications: Tuning for Performance and Scalability, Second Edition. Just click on the titles, they are links.
Topics as the following (in those documents) can be interesting for you:
We cannot forget that the Web Applications are just a client to the data sources, such as the metadata, and the WIP database. If the number of possible connections on the Web Application Server are more than the number of possible connections on the WIP database, it can occur than the WIP databse won't allow more connections from the Web Application Server.
Something else, and just in case it helps: SAS Web Infrastructure Platform Data Server (or a third-party vendor database). The web application server is configured to test the database when it provides a new connection from the connection pool. The checks occur, at most, every 30 seconds. As a result, the web application server can recover from a failover or restart of the database but can experience up to 30 seconds of trouble connecting to the database before it recovers.
Just to finish: indeed an update of the basic platform to M3 I think it is a good idea. M3 brings a lot of improvements related to platform stability and performance, which is exactly your problem.
I hope it can help you. Please let us know how this progress.
06-24-2016 08:10 PM - edited 06-24-2016 08:14 PM
We've been running SAS9.4M2 BI server for nearly two years now and once we got a few troublesome problems sorted at the start it has been working reliably ever since. Our experience is that this release is fine as long as you diagnose and fix any problems early.
Our biggest issues were with Environment Manager and ensuring that when the SAS servers are stopped and started that everything happens in the right order. Unfortunately this area is not well documented so there is a fair amount of trial and error and getting SAS involved to get things fixed.
I guess you know about the SAS9.4M2 slow web server start problem (at least 30 - 40 mins!). We also found that unless you start/stop Env Mgr agents correctly it can end up generating huge logs on your web server and using most of the CPU.
We did plan to upgrade our M2 SAS BI to M3 so we could add SAS VA. In the end we have not done that and added a separate SAS VA 9.4M3 instead. This is because it was going to take a week's work to do the upgrade and we couldn't afford SAS to be down for so long. This is another weakness in the current SAS architecture - it takes too much time and effort to apply maintenance mainly for the web server. I have heard that SAS are working on improving this but I don't know when this will happen. Env Mgr is much improved in M3 and the web server start is way faster.
Hope this helps.
06-27-2016 04:44 AM
Thank you SASKiwi
I find your posts very insightful. We are currently looking at a few other Visual Analytics solutions as well.
Yes - the JVM on SASServer1_1 takes about 25 - 35 minutes to start. And I run Batch script which SAS gave us to stop and start all of our Web App JVMs. We have a Load Balanced Web App Server cluster with 2 nodes, and I have to stop and start both of these nodes, and it takes me a good 60 - 90 minutes to do this.
SAS indicated that in SAS9.4M3 they reduced this down to 5-10 mins. I am about to give up on EM at the moment. It is off most of the time now and a real pain to figure out how to stabilise it. So it is Trial and error diagnostics for me at the moment. Hope you have a good week ahead!
06-27-2016 03:37 PM
Thank you @Lenvdb for the feedback. Sometimes it is hard to know if your answers will be helpful or not. I can imagine a GRID setup like yours will be harder to manage than our non-GRID one. With the Web Tier I found it really useful to check out the various web server logs as that was often the key to figuring out what was going on.
Env Mgr isn't essential for monitoring SAS server but is definitely a nice to have. You can use other tools like Perfmon for example.