Architecting, installing and maintaining your SAS environment

Server performance slow down?

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 19
Accepted Solution

Server performance slow down?

Hi There,

I've migrated our servers from SAS 9.3 (EG 5.1) to SAS 9.4 (EG 7.1) with 10-clients & two Virtual servers (VMWareVSphere): 1. Metadataserver (4-cpu, 16gb RAM) & 2. Mid-tier has Web infrastructure platform data/object spawner/share/connect/DIP Job runner/JMS Broker/Cache locator/WbAppServer &Environmanager server/services (8-CPU, 32 GB RAM). Migration was successful.

When clients (2-4 at the same time) try to run codes/query from EG7.1 or tries to connect the connection/run is extremely slow.

I've asked SAS Technical support and fine-tune all servers according to their suggestions & even fixed the networking issue with our IT, still we face sluggish connections.

Is there any suggestion/solution to this issue?


Accepted Solutions
Solution
‎08-21-2015 11:32 AM
Occasional Contributor
Posts: 19

Re: Server performance slow down?

Hi All,

Thank you very much for all your suggestions and support. I did install fresh SAS-Servers on New Windows2012R2 Servers and now even with 4-cores/16G RAM whole system works very fast and smooth.

Hitesh

View solution in original post


All Replies
Super User
Posts: 3,261

Re: Server performance slow down?

Posted in reply to devlekhu_98

Diagnosing performance issues can be tricky and take a long time to resolve because there are so many possible causes.

I would start by checking the servers for CPU, memory and IO performance - are they maxing out? I'm assuming you also have SAS Application servers running - are they on your server 2?

If the servers are running OK then possibly network connections might be slow. Run a network analyser to see if there are any bottlenecks.

Benchmark a SAS program running from EG, then run the same program in batch on the application server. Are there any performance differences?

Trusted Advisor
Posts: 3,215

Re: Server performance slow down?

Posted in reply to devlekhu_98

You didn't mention

- The used OS, Can be Windows or a Linux (*Unix) type.

- Where the SASapp servers are configure and run

- How the authentication is done. Host based as most reliable or some tricky one using shared accounts or IWA

- What has done at the VM virtualization level.

- The network access layer to those machines (firewalls&speed)


Peforrmance&tuning is a difficult topic as it is requiring understanding of all the involved  interactions. This is more complicated with virtualization.

There is not much on that with SAS. If you want the best out of your hardware the bare metal is the limitation. Licensing is another issue, you could come into paying for all the hardware where you are expecting the virtualized limit is the one having being agreed on. Check that with your legal affairs (procurement).   

Two nice papers:

- http://www.vmware.com/files/pdf/techpaper/vmw-sas-moving-virtual-environment.pdf 

- http://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5.pdf

The first nasty one is having a good communication with the VM-guys. Your OS with SAS is just an application for them. They can have ideas of common application settings that are not a fit with SAS usage.

The firewalls&speed can become an issue (note the NIC VM setting) but that is more likely not the biggest one. 

To eliminate that one can be difficult (needing measurements). The first step is having an other RDP ore terminal you can work on the machines seeing what is happening.

"Well with one user working it is looking to work well. That is your installation verification all well project successful. Two or more processes and that is not performing well? User isse not the installation/configuration." Here you have your challenge in expectations and trying solving those is a hard job.

You are saying two things:

1-  When clients (2-4 at the same time) try to run codes/query from EG7.1 it runs slow  (processing  IO - cpu)

2-  When trying to connect. start run from EG7.1  it is slow (metadata login and start a sas process on the os level) . 

Issue1 is a resource CPU/IO/Memory issue not able to deliver sufficient of those by the VM for each of your sas processes.
As your are talking about Eguide, the midtier is not involved. Only the metadataserver and the app-server more dedicated the WS are the ones to look at.

What is happening on the OS-level that is holding up your sas-processes.   It could be thrashing (memory constraints) IO limits or concurrent process limits (cpu).

      

Issue2 can be a resource issue but also other things. As some resource issue the metadataserver can be suffering from lack of resources so that one becoming a problem to deliver fast.

The start (login) does several things that depends on how authentication is done. The metadataserver is running a process on a very high privilege level (unix sasauth) for that. Whtn that authentication process is suffering from something the logins will be delayed. You could eliminate that by defining an internal user and using SMC or something like PLM with that.  When they behave well but the login EGuide is problematic than there is an internal SAS issue with the login and the following start of a OS sub-process.

These days an invalid login is followed with a delay for a retry. What if that delay is done on behalf of the metdataserver generic process not the involved user proces. That one can be seen be checking the login steps with 39891 - Using PROC PERMTEST to diagnose UNIX host authentication issues in SAS® 9.2 and later releas... 

---->-- ja karman --<-----
Occasional Contributor
Posts: 19

Re: Server performance slow down?

Hi Jaap,

The used OS is Windows 2008 R2 on both VM-servers. I've migrated system to 9.4 from 9.3 in January, 2015. We had 2-servers (Metadata & SASApp) in SAS9.3 system. The old system was running faster. We received a plan file (Metadata & Mid-tier with SASApp) to migrate the system and migrated successfully. SASApp servers are configured on Mid-tier server. I only have two VM Servers, 1. Metadata server 4-cpus, 16gb RAM and 1. Mid-tier/SASApp 4-CPUs, 32GB RAM.

Authentication is host-based.

The other two-topics (VM & network access), I don't have complete knowledge.

In all the system was working with old SAS9.3 system!

Super User
Posts: 3,261

Re: Server performance slow down?

Posted in reply to devlekhu_98

We just went through a very similar exercise migrating from 9.3 to 9.4 on Windows VMWare servers. The only significant difference is we have 3 servers: metadata, app, and mid-tier. Our 9.4 performance is very similar to 9.3.

One thing we did see with 9.4 is the mid-tier server is much more heavily used than in 9.3. Much of this appears to be Environment Manager monitoring. It runs constantly with about 25% CPU and 13-14GB memory used.

I would look closely at your combined mid-tier / app server performance as it doesn't appear to be a good idea to mix the two if you've also got Environment Manager monitoring switched on.

Occasional Contributor
Posts: 19

Re: Server performance slow down?

Hi SASKiwi,

We had old SAS 9.3 plan with only two servers (Meta & App). Time of migration I asked SAS to provide me with a plan file to migrate and they provided 2-servers plan. The didn't recommend additional server plan. My mid-tier has App & All mid-tiers. It runs with almost same 20-25%CPU use (Goes upto 80-95% with few jobs submissions) but with only 2-4GB RAM (Lower than yours).We are observing a weird pattern of slowing down as in the morning system runs faster even with 2-4 clients login but in afternoon it runs extremely slow (3 minute job takes almost 2-3 hours to run!). Our network team says something (Most likely JAVA processes) holds resources and releases them very slowly (After office hours & Night enough to release completely). I've couple of questions:

1. Can I stop this environment manager ? or Even SASWebApp Server ? What's minimum set/subset of servers/services I can run with? Do I need to run DIP Job runner/JMS Broker/SHARE/Cache locator/Deployment agent? I ask these question to SAS Tech-support but they said I need to run all!

2. Did you followed fine-tuning document from SAS?

3. What resources allocated to all 3-servers (CPUs/RAM) etc?

Thanks

Super User
Posts: 3,261

Re: Server performance slow down?

Posted in reply to devlekhu_98

This SAS note explains how to stop and start the Environment Manager agents. You must use the BAT files for this - if you try just stopping and starting the Windows service SAS will start writing huge amounts of errors to server logs degrading performance. They run on all servers and on each 'level' you have in your installation. There is also Environment Manager monitor itself on the mid-tier. I'll check how that gets stopped.

http://support.sas.com/kb/52/668.html

Our app and mid-tier servers are both 4-core, 16GB.

It sounds like something really extreme is happening to degrade your performance. I would also check out your mid-tier server logs in case they are generating a lot of errors.

Occasional Contributor
Posts: 19

Re: Server performance slow down?

If I stop the Environment manager, is going to affect any functionality?

Super User
Posts: 3,261

Re: Server performance slow down?

Posted in reply to devlekhu_98

If you only stop the Environment Manager agents then you are only stopping monitoring from those servers. Environment Manager itself will still work. This doesn't impact your metadata and app servers.

Occasional Contributor
Posts: 19

Re: Server performance slow down?

Thanks I'll stop that and monitor performance. If you want to see logs please let me know the name of the server and I can provide you with log of respective server.

Thank you

Super User
Posts: 3,261

Re: Server performance slow down?

Posted in reply to devlekhu_98

I suggest you try looking at the mid-tier server logs yourself to start with. We have had issues with these filling up with continual errors degrading performance so I was wondering if you were having the same issue. The actual errors themselves aren't very useful, it is more the fact there are so many of them that the logs grow massively to many GB in size.

Occasional Contributor
Posts: 19

Re: Server performance slow down?

Is there any particular log or logs for all servers running on mid-tier? Any particular folder? My Lev1 folder has 12.1GB size (Most of them are "EAR Files" &  in "Lev1\Web\Staging" directory), looks like you are right. Do I need to delete those old logs manually or is there any way we can limit number of logs? Do you have any idea about size of that folder?

Should I stop WebApp Server? Does it affect any SAS-Functionality? My observation is: When I start this server, it runs JAVA.EXE process and starts using lot's of CPUs.

Thanks

Super User
Posts: 3,261

Re: Server performance slow down?

Posted in reply to devlekhu_98

Have a look here. If the logs are huge they can be deleted and then its easier to see as they start to grow again if they are filling with errors.

\\MySASWebServer\SAS\Config\Lev1\Web\Logs\SASServer1_1

Solution
‎08-21-2015 11:32 AM
Occasional Contributor
Posts: 19

Re: Server performance slow down?

Hi All,

Thank you very much for all your suggestions and support. I did install fresh SAS-Servers on New Windows2012R2 Servers and now even with 4-cores/16G RAM whole system works very fast and smooth.

Hitesh

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 13 replies
  • 4846 views
  • 7 likes
  • 3 in conversation