topic Re: Abundant network traffic between metadata server and management console in Administration and Deployment

Abundant network traffic between metadata server and management console

jklaverstijn — Tue, 30 Oct 2018 14:55:34 GMT

Dear fellow admins,

We are in the process of migrating a number of departments from their own dedicated SAS server to a large common grid based environment. This has resulted in a larger consolidated set of metadata that is now being served by a cluster of 4 metadata server nodes. Everything on the server side is Linux based. These servers have lots of memory and CPU load is never high. The nodes talk to each other over a 10GB connection.

We see an increase in the startup time of management console that seems to be proportional to the volume of metadata but also very dependent upon network conditions. Users that are connected via a VDI (virtual desktop) are the lucky ones: 64 seconds before getting control over the interface. Worst of are the Wifi users (also the majority) with 145 seconds. No wonder people start complaining.

We have done a network trace and see a large amount (130k) of relatively small (avg. 470 bytes) number of tcp packages. So it makes sense that network latency is our worst enemy. Total volume is not much (58MB) but is being delivered in very small chunks.

I was wondering if we are the only ones that see this and, if not, how this can be dealt with. We will shortly contact SAS Support as well but I was wondering what's happening in the field.

So, any experiences you would like to share?

Many thanks in advance,

-- Jan.

Re: Abundant network traffic between metadata server and management console

nhvdwalt — Tue, 30 Oct 2018 15:01:56 GMT

Hi @jklaverstijn

Do this test....

1.) Pick a regular user and measure the response time as a baseline.. like what you've described

2.) Then, take away all the user's permissions. Easiest is to remove all the user's groups and assuming you are not granting lots of permissions to SASUSER group. Redo the test and see what the difference is.

If the response times are a lot better, it might means that your user has permissions to many metadata objects.

Re: Abundant network traffic between metadata server and management console

JuanS_OCS — Tue, 30 Oct 2018 15:07:37 GMT

Hello @jklaverstijn, Jan,

I am not really sure if something that I already experienced, it is also what you are experiencing...

Under my experiences, this SAS Note has helped me great deal in the past :

Problem Note 49308: SAS® Management Console is slow to launch the first time http://support.sas.com/kb/49/308.html

I know it is not mentioning anything about networking, but I wonder about those small packages traveling along your net.

Do you have any local installation of SAS Foundation, perhaps on a VDI? TKJNI_OPT_TRACE can help a lot with tracing the java TK components.

Do you experience the same also with a SAS Base metabrowse or a connection to SAS DI?

Re: Abundant network traffic between metadata server and management console

nhvdwalt — Tue, 30 Oct 2018 15:11:34 GMT

@JuanS_OCS makes a good point.

If you are running UNIX, you could also start SAS Management Console on the UNIX command line through X11 and test the effects of any AV's on Windows.

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Tue, 30 Oct 2018 15:21:36 GMT

Good one @nhvdwalt. We did that and see no reduction in the amount of packages but the average size drops to 200 bytes! And we passed this test for the tightness of our authorisation model 🙂

There is no noticable improvement in sasmc start time. As the number of packages is the same, this confirms that latency is the killer here.

Thanks,

- Jan.

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Tue, 30 Oct 2018 15:24:05 GMT

Thanks @JuanS_OCS

Yep, did that and X11 is approximately the same as the VDI. But this is not available to end users, just admins do this.

Thanks,

- Jan.

Re: Abundant network traffic between metadata server and management console

JuanS_OCS — Tue, 30 Oct 2018 15:26:31 GMT

@nhvdwalt, @jklaverstijn,

I did not mention anything about X11. Nice idea, but not mine, it is @nhvdwalt who introduced it.

Jan, did you have the chance to give read through the SAS Note?

Re: Abundant network traffic between metadata server and management console

Kurt_Bremser — Tue, 30 Oct 2018 16:07:14 GMT

To remove the network from the equation, start SMC via X11 on a metadata server itself.

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Tue, 30 Oct 2018 16:10:57 GMT

Ah yes, getting things mixed up here. Sorry for that 😉

Yes I went through the doc. Most of it was not relevant. But the test with TKJNI_OPT_TRACE looks interesting but also time consuming. I will have to get back to you on that. It's good to know it as it is also an early warning for what SAS Support may start with. Better be prepared.

Thanks,

-- Jan.

Re: Abundant network traffic between metadata server and management console

nhvdwalt — Wed, 31 Oct 2018 05:08:56 GMT

@Kurt_Bremser suggested a good test.

Then, try two tests, one with the server's actual name and another one with localhost.

Have you looked at the actual network path from VDI to the Metadata server ? Some enterprise networks can be really complex.

Re: Abundant network traffic between metadata server and management console

PaulHomes — Wed, 31 Oct 2018 06:17:05 GMT

Hi Jan,

A few things of the top of my head....

Any there interesting messages in the SAS Management Console client log SASMCErrorLog.txt?

The worst case of slow SAS MC start I've encountered was due to anti-virus/malware scanning. Any chance of temporarily disabling it to see how much impact it is having?

Is it just SAS Management Console? Do any other Java based SAS apps connecting to metadata have the same issue (SAS Data Integration Studio, SAS Information Map Studio etc)?

Is DNS resolution working well (dig, nslookup etc)? I have seen long delays in the past with slow DNS and timeouts.

SAS Management Console also makes a connection to the mid-tier during startup/login - what sort of mid-tier response are you getting from those clients?

Perhaps if you are able to temporarily run the metadata cluster in non-cluster mode you could see if there is any differences there with respect to metadata server cluster node redirections?

Cheers
Paul

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Wed, 31 Oct 2018 10:00:15 GMT

Hi Paul,

Other clients are okay. Opening a job in DIS is considered sluggish but not alarmingly so.

We have covered the other aspects you mention like DNS and anti-virus. Nothing to be gained there.

Tha pathj we are now floowing is the number of projecty repsitories. As part of our AD synchronisation code we also create a project repository for every new user. This seems to play a big role in startup times of both the server itself (which makes perfect sense) but also SASMC. In a DEV domain I have used a script to create an arbitrary number of project repositories and have times startup of sasmc as a function of that number. And lo and behold, the correlation is strong. 10 repositories results in a 40 seconds startup time. This linerarly scales up to 5min15 for 200 repositories. In reality we have 350! All as sasadm@saspw.

So yeah, we are up to something here.

I am now running a single node (startNoCluster) and trace logging turned on. I will see what's happening, especially for unauthorized users who have no business in other's project repos.

More to follow.

Thanks,

-- Jan.

Re: Abundant network traffic between metadata server and management console

nhvdwalt — Wed, 31 Oct 2018 10:19:54 GMT

When last did you do a metadata analyse/repair ?

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Wed, 31 Oct 2018 12:47:59 GMT

We do that on a regular basis. Not scheduled as we do not have scheduled down times, but whenever we're down for other reasons. Last time was a week ago when we ran some reuqired security patches for Linux.

Other than that we would give an arm and a leg for on-line analyse&repair. Down time is a bit of a no-go; that's why we have a cluster in the first place.

-- Jan

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Wed, 31 Oct 2018 12:55:42 GMT

Fitst of all thanks for all the valuable input. It did help with eliminating much of the environmental contributors to metadata headaches. The plot is thickening however! As said I have created 200 project repositories and there sasmc and/or mds seem to scale poorly. I have set logging to trace (especially the App.IOM logger) and see that every one of those 200 repositories is queried 25 times! So 5000 lines like this before sasmc even becomes responsive:

outMetadata=<RepositoryBase ID="A0000001.blah" ... Name="testreposxxxxx" ... >...</RepositoryBase>

It is also clear fro those logs that sasmc is querying for this 25 times for each project repos. Seems crazy to me.

I think we have enough for a call to tech support.

I'll keep y'all posted.

Cheers, Jan.

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Wed, 21 Nov 2018 15:01:28 GMT

Intermediate update: we have deleted many of the project repositories after careful selection. Going from 320 to 45 made a world of difference. We're not there yet and expect more from our track with tech support, but wanted you to know this.

There's a gotcha: when you do this metadata clients like sasmc can/will get very upset. They start complaining about missing repositories. Probably harmless but very confusing to the users so we moved this to outside office hours. To avoid endless mouseclicking we used a script with sas-delete-repository command line utility with a short sleep between every cycle to keep it light for the metadata server cluster.

Sooo we're in a much better shape now. But the in the track SAS started with questioning the use of project repositories in the first place. Couldn't we just [...]? Really, SAS?

Regards,
-- Jan.

Re: Abundant network traffic between metadata server and management console

Kurt_Bremser — Wed, 21 Nov 2018 15:08:14 GMT

I guess the mechanism for separate repositories was meant to cover something like 2,3,4, maybe 5. Since it was never meant to be used for hundreds of repositories, nobody tested that. Or did not care for performance penalties incurred.

Re: Abundant network traffic between metadata server and management console

PaulHomes — Wed, 21 Nov 2018 20:41:57 GMT

Thanks for providing the update Jan. Purely out of interest, what is the use case for so many project repositories? Is it simply that there are lots of SAS DI Studio users doing change managed updates or are you you using them in another capacity?

Re: Abundant network traffic between metadata server and management console

jklaverstijn — Wed, 21 Nov 2018 21:27:10 GMT

There was no purpose for the 300+ repositories, that's why we managed to
eliminatie so many. But 50+ (and more to come) is for real. We are a big
shop. Real big. And most teams employ change management for their
development process. Our scale is why we use grid manager and clustered and
load balanced everything.