EG8.2 - high memory usage while browsing libraries with the server-list, anyone else noticed?

[ccaero]

Hi all,

currently we are using EG 7.15 (HF8) in production for all users. From time to time I check, if EG 8.x is suitable for production in our environment. Up to EG8.2 update 2, I was't happy with the performance compared to EG 7.15 - especially delay while opening tables or browsing through the server-list. Now with update #4, the overall performance is much better.

But now, I noticed another performance-issue:
Every-time i "open" a library in the server-list and let me show all tables inside (click on more) the applications memory-usage strongly increases - of course depending on the count of tables inside the library.

For example:
We have some large libraries (1000+ tables per library), the biggest counts about 7000 tables. If I open this very large library in the server-list and click "show all" the memory-usage rises up to 800MB (checked with windows task-manager). If I than open only 3-4 libraries (with 1000+) tables in the 32bit version of EG, the memory-usage rises above 1GB and i get an application-error - the EG-log show "outofmemory" Exceptions. If I using the 64bit version of EG i can easly push the memory-usage up to 4GB - just by browsing through! Of course the 64bit version doesn't crash, but this high memory-consumption isn't acceptable for me and I prefare to use the 32bit-version because it supports more import/export-dataformats.

In EG 7.15 HF8 (32bit & 64bit) I never saw such a high memory-consumption. When I do exacly the same in EG 7.15 like in EG 8.2, the memory-usage stays below 200MB - sometimes the memory-usage even decreses in EG 7.15 while browsing through the server-list (better garbage collection?).

If I using the "File -> Open -> Data"-dialog in EG 8.2 for browsing libraries instead of the server-list, the memory-usage stays low.

Did anyone else notice this kind of high memory-usage in EG 8.2?

[andreas_lds]

This could be a bug, have to talked to tech support about it?

[ChrisHemedinger]

Just last week there was another update (EG 8.2 update 4) to address some SAS library loading issues.  I recommend trying that to see if it improves.

[SASKiwi]

@ccaero  - With EG 8.2 Update 4 I'm seeing exactly the same memory behaviour as you. Every Server List library you expand increases memory usage and if you have thousands of datasets like we do then memory usage increases to several GB. Collapsing libraries doesn't free up memory. Even closing the server connection doesn't free up memory. Reconnecting to your SAS server finally does release the memory but it starts increasing again as soon as you expand libraries.

If you haven't raised a Tech Support Track for this yet then I'd definitely recommend you do so. I'm going to report this as well.

This is what Windows Task Manager looks like after opening a few large libraries - memory usage is over 2GB!

[CaseySmith]

There is a known issue that was (unintentionally) introduced in EG 8.1 that causes the memory usage to grow linearly based on the number of items displayed in the Servers tree.  (The tree control we are using is, unfortunately, not fully virtualizing.)  For libraries with lots of data sets, or folders with lots of files, this can quickly lead to an out-of-memory condition.  To try to avoid this issue, we limit the number of child items we will show in the tree to 500 by default.  However, there are still "More" and "All" links that will allow you to load more than 500, which has the side-effect of making it easy for a user to still encounter this out-of-memory condition.  Since clicking the "More" and "All" links can potentially load lots more items into the tree, thus lead to the out-of-memory condition, we recommend using the File Open dialog to navigate for additional items (beyond 500) rather than using the "More" and "All" links.

To make it less likely for users to encounter this issue, we replaced (in upcoming 8.3 release) the "More" and "All" links with "Browse".  For example, if you scroll through a library in the Servers pane that has more than 500 tables, you'll now see something like this...

Then, you can click the Browse button to bring up the file open dialog to navigate to the rest.  (In fact, it probably makes more sense to navigate in a file open dialog when there are lots of items, rather than in a tree control, but the reason we put it there is to help alleviate this issue.)  (Before 8.3, we showed "More" and "All" links here, which would still allow users to attempt to load more/all items, thus greater potential for encountering the out-of-memory condition.)

While this change makes it less likely for EG to consume an exorbitant amount of memory, memory consumption will still grow the more items you show in the tree by expanding large libraries/folders.  If it is an issue, again, we recommend (as a workaround) using the File Open dialog to navigate large libraries/folders instead of the Servers pane.

Casey

[ccaero]

@SASKiwi

Thanks for sharing your experience, so I'm not alone with this issue. I haven't reported the problem to sas support yet, but I will do that.

@CaseySmith 
I'm using EG8.2 Update 4 and I do not see the "browse" link in the server-list. I still see the "More" and "All" Links:

But for me, the "browse" link is not really a solution for that problem, it is more an work-around. Browsing through all avaible libraries via the server-list is more comfortable than using the "browse"-dialog. So for me the serverlist is "broken" in EG8.x - in EG7.15 the serverlist works really nice and fast, even with very large libraries.

Here you can see the difference between EG8.2U4 and EG7.15HF8 in terms of memory usage -> the same libraries are opened in both tasks:

I will send you an link to a screenrecord-video via private-message where you can clearly see the issue and the difference between EG8.2U4 and EG7.15HF8.

@ChrisHemedinger 
As I said in my opening post, I'm already using Update # 4 🙂

[CaseySmith]

@ccaero, I apologize, I made a mistake...  the change we made to replace "More" and "All" with "Browse" was made in our upcoming EG 8.3 release, not currently in an 8.2 Update, which explains why you do not see it.  I edited my earlier post to correct it.

You are correct...  the "Browse" button is a workaround, not a solution.  The solution is to replace the treeview control we are using with a fully-virtualizing treeview control.  However, that is a non-trivial amount of effort and risk -- not something we can do in an update.  For now, we have to live with the workaround (use the File Open dialog in favor of the "More" and "All" links to explore library members past the first 500), however we do hope to have an opportunity to provide a full solution in a future release.  Sorry for the inconvenience.

Casey