Enterprise Guide performance over a slow network

[Nigel_Pain]

Like many organisations, we're primarily working from home at the moment, connecting into our corporate network over a VPN. Our network folk have massively increased the pipe into the network to cope with this, but network performance is probably mostly limited by the speed of users' own home networks. I have an Enterprise Guide project (7.15, connecting to 9.4M5) with a process flow of around 16 task or code items which I run regularly and I've been finding that it is taking maybe 10 to 15 minutes to run (rather than a couple of minutes when in the office). On the other hand, I tried exporting all the code in the flow to a program, and ran that in EG. This way it took less than a minute.

I'm assuming that there's an overhead in uploading the code from each item in the flow, and possibly more so with tasks than programs. Am I correct in this?

I've spent considerable energy (with variable success) trying to persuade users to use process flows to their full extent, breaking up large blocks of code into programs or tasks containing single steps so that it's easier to manage and debug. If I now advise them to go back to monolithic code I'm in danger of undoing all that work!

[ballardw]

Are your projects on your computer at home or on the server?

I would expect if the project were on the server then everything except possibly final output would run pretty much the same as when in the office.

[Nigel_Pain]

The projects are stored in various places, mainly shared drives, but they are opened and run in Enterprise Guide on users' laptops. We don't have a way for users to work directly off the server.

[ChrisHemedinger]

ODS results are one source of latency that becomes exaggerated on a slow network.  If you can change your results options to use a more efficient result type, things might be snappier.

• If doing traditional SAS/GRAPH graphs, definitely don't use ActiveX (used to be the default). Try PNG or JPG.
• If you can by with just text output, try that (ODS LISTING).
• Play with HTML5 (EG 8.1/8.2 default) and using SVG vs PNG for ODS graphics (SGPLOT, etc).

[Nigel_Pain]

Thanks @ChrisHemedinger, always useful to get information from the horse's mouth (not that I'm suggesting...)!

There are a few good things to try here, and I definitely have a few objects with ActiveX output so I'll try changing these to PNG. Unfortunately, being still with 7.15, HTML5 and SVG don't appear to be options. We have a freeze on system changes at the moment but when things get back closer to normal we are planning an upgrade to M6, mainly to escape Flash in VA 7.4, but that will also include the latest version of EG, I would imagine. Having briefly skimmed through a "what's new" posting here, I'm quite excited by that.

Interestingly, it's not just output which is the problem. My first step is an import of a CSV, using PROC IMPORT in code from a shared drive. This is also very slow to get going. It sits for a minute or two with the message "preparing the server..." in the Task Status.

[ChrisHemedinger]

When you say "PROC IMPORT" do you mean a proper code-based import, or the Import Data task? If the latter, you should check the performance options in the task to skip the data "cleanse" and limit the scan.

Even better, if you're willing, use the Copy Files task to copy the file to the server and then run PROC IMPORT directly from there.  This will allow you to separate actions for data transfer and the subsequent import.

[MCCL1250]

I am also interested in this conversation as I too am having this issue. 

We are running on EG 7.15 and connecting to the server (9.4M5) and am seeing much the same thing.

Things are running much smoother after hours when there is not a lot of usage to the VPN.

But we cannot run all things off hours.

I am seeing this same "preparing the server" being a frequent hang-up. Also if there is a condition on the execution of the node in EG, that seems to take a lot longer than it should.

And exports and "download" or "uploads" seem to be taking a lot of time. 

I did turn off the display results option, and that is speeding the development work up. But other things can take 2-5 times as long to run.

Any help or advice would be appreciated since it looks like we may be in this situation more long term.

[Nigel_Pain]

By PROC IMPORT, I mean the code. I originally started off with an import task in the project but converted that to straight code. In fact, it's a modification of the data step code generated by the EFI which I've used rather than PROC IMPORT itself.

The file itself is in a shared drive on our network, so there shouldn't be any extra latency in importing it from there, as it's on the same subnet as our compute server.

Some useful/interesting comments from @Kurt_Bremser and @SASKiwi below.

[SASKiwi]

@Nigel_Pain - I suggest you copy a representative EG project onto your laptop hard drive and run it from there. Is it a lot faster than running off a shared drive or not?

I'm in a similar boat to you, working from home over VPN. However we develop mostly in SAS code on EG 8.2 and that performs pretty well as long as VPN isn't overloaded. I would expect running projects to be slower than running code.

I'm also lucky to be able to remotely connect into a PC hosted internally on the company intranet. Not surprisingly that gives you the same performance that you would get in the office. So that is really the best solution for slow networks - have EG hosted internally within your company's intranet either on PCs (virtual or real) or as a Citrix gateway application.

[Nigel_Pain]

Some interesting points here. I'm not sure how the location that the project is saved would affect its performance. I'm opening it on my laptop (so I can see that there might be a performance issue in actually doing this) but once it's open, it's running from my laptop, wherever it has been saved (or am I wrong about that?)

Unfortunately, connecting into remote desktops isn't an option for us, as desirable as it might be!

Thanks for your advice.

[SASKiwi]

@Nigel_Pain - I was just wondering whether reading and writing the project from or to a shared drive causes any of the slowness, that's all. A test on your local drive would prove that one way or the other. 

[Nigel_Pain]

Not really seeing any noticeable slowness opening projects from a shared drive.

[Kurt_Bremser]

EG is very "talkative" over the network, with a lot of back-and-forth for every single operation. This becomes very obvious once the network latencies grow. I have the same long wait for "preparing the server" that you do (same situation as you, VPN over internet to a private ADSL at home), and equally long waits for the results to display.

While developing, I work in small steps, and so take my time. But for "production" purposes, codes are run in batch or quasi-batch from one code node.

Since I am also a pure coder and develop for batch jobs, "developing" means working in a code node and running single steps selectively.

From this POV, I can only suggest that once a project of yours has reached "production" stage, you export all code to a single code node and run that.

[Kurt_Bremser]

Oh yes, to clarify: the choke point is not the network speed as such (the amount of data EG fetches when displaying a dataset is actually minuscule compared to all the **** you get from any of today's websites which are overloaded with unnecessary junk), but the latencies introduced by the many hops (firewalls and proxies) in between.

In-house network: 1 ms or less, VPN: 40 ms (just measured here)

So a transfer needing 20 packets will need ~40 ms (packet + ACK) in your company, but 1600 ms (1.6 seconds) at home to finish. And that's what you (and we all) experience.

[Nigel_Pain]

That's interesting what you say about the hops being the source of the latency, Kurt. It makes sense given that, as you say, the amount of data being uploaded in a piece of code is pretty small. I have advised users to turn off opening data and output on task completion, and also opening data when it's added to a project.

Being basically a sysadmin, and in SAS terms a platform admin, I don't do a lot of developing myself - the project I was describing is one instance, where I'm actually monitoring and trying to forecast disk usage on a completely separate system. But we have a considerable number of analysts, mainly statisticians, economists and data scientists, for whom EG is their main method of working in SAS. It's these people who I've been trying to advise on the way they work. It's mostly interactive: there's very little batch running.