Performance differences: SAS/Server vs SAS/PC

[deleted_user]

EG will consume memory on the workstation.

When EG connects to a remote server, a SAS session is started which will consume some memory there.

I recommend you start with -memsize 0 in the configuration file for remote servers, and then, if there are any "out of memory" issues, investigate them, and either start to limit the memory, increase the memory, or resolve the software process problem.

I would also recommend that you read the online SAS documentation extensively and thoroughly.

[deleted_user]

Thanks. Thats good advice I will heed. This is probably my 12th day exposed to SAS. So far, I've consumed SAS for dummies, some of the white papers for running SAS on wintel, and halfway through sas companion. Its still alot to juggle in my head, but I find myself getting fascinated with SAS

-Josh

[deleted_user]

You are welcome.

One word of caution about the SAS Online Documentation. While it is extensive, and mostly complete, it can also be rather convoluted in where stuff is actually stored. Many times, I have followed embedded links to other pages and have no clue has to how to get back to them later directly, without having to follow the same link to link to link path I used originally.

Also, remember, with computer systems, the details matter. So once you have read the general discussions about what SAS can do, read the detailed command/statement/option documentation.

I have also found that sometimes the generic discussions have excellent and useful samples -- ?stealable? no ! re-useable ! code -- that shows something that isn't clear in the detailed doc. Other times, the detailed doc does a better job of explaining the generics and provides better samples than the generic discussions.

As a friend of mine from ages past would say:

"Be careless, but above all, enjoy".

[deleted_user]

I've experienced that first hand with -memsize. SAS Support advised us that setting it to 0 or max was a bad idea. But SAS 9.1 companion for windows states in Page 523 that "For optimal performance, use the default value of 0 for MEMSIZE". The formula used in some white papers for NT makes sense to me
total physical mem - 512/ Total number of concurrent sessions ;if memsize actually pre-allocates that memory range just for one specific session.

[Kenny]

Yes, my comments regarding memsize=0 were motivated by SAS Tech Support. Since each site is different, who can say what a tech support person might tell one person over another. And I don't understand why the default sasv9.cfg file has memsize=64M shipped from SAS if they recommend 0???

I also assumed that we were experiencing problems that may actually have been non-existent according to Chucks comments on memory usage for individual server connections. I would have to investigate that again.

I wonder if the advice from SAS Tech support were based on real world experience over a broader range of usage - in other words, maybe Chuck's shop has better trained programmers/analysts who don't attempt to summarize 40 million rows by the key field! 🙂

Kenny

Message was edited by: Kenny Message was edited by: Kenny

[deleted_user]

40 Million?

Where I was at I was summarizing > 1 Billion daily observations by their 1300ish to 1500ish and 250,000ish "key" values, every day. Then there was the other data source that was 10's of million daily records, and about 5,000 to 10,000 "keys".
the number of produced summary variables varied from about 5 to about 34.

The wonderful world of equity options trading (and quoting, the real gut buster -- transaction rates in bursts > 40,000/second, and you had to process them all with < .5 second response times) Message was edited by: Chuck

[deleted_user]

I wonder what your server physical specs are. You must have alot of physical memory to crunch that much data, or your team must have done a great job optimising your data paths to minimise IO and network latency

[deleted_user]

Like I said earlier, I had been misinterpreting the memory utilization statistics for an AIX server and increased the memory to 32 GB. At that point, with memory utilization still being reported at 100% I discovered my mistake, and found that SAS was using < 4 GB for all us 20+ concurrent users + OS + monitors + ...


At the time, I was the defacto SAS admin, and so had control over the configuration settings: -memsize 0, -bufsize 32k, -bufno 3, -asynchio, -threads. I don't remember what we set -sortsize to.

SAS generally only allocates memory for the buffers. So, if you create 12 datasets in a process, that's 12 * 3 * 32k < 2 MB + some other insidental memory.


Current employment, Daily processing example of memory usage:

at least 13 jobs each day

[pre]

on 1 2 3 4 5 6 7 8 9 10 11 12 13
off 1 2 3 4 5 6 7 10 9 8 12 13 11



largest dataset
job memory consumption observations variables
1 278,506 kB 3,944,614 29
2 283,890 kB 7,519,583 38
3 166,183 kB 6,549,492 96
4 275,000 kB 12,687,208 40
5 37,973 kB 43,815 24
6 51,161 kB 170,777 8
7 22,611 kB 43,515 51
8 269,938 kB 2,292,277 18
9 268,035 kB 674,705 21
10 129,098 kB 337,333 15
11 537,220 kB 8,981,167 76 (contains a select distinct for 22 variables)
12 21,988 kB 143,198 4
13 2,978 kB 162 24


jobs 5-11 were concurrently running for a time.
I am not the SAS admin ...



SORTSIZE=256M Size parameter for sort


MAXMEMQUERY=0 Maximum amount of memory returned when inquiring as to available space
MEMBLKSZ=16777216 Size of memory blocks allocated to support MEMLIB and MEMCACHE options.
MEMCACHE=0 Activates use of Intel Extended Server Memory Architecture memory for file caching.
NOMEMLIB Disable use of Intel Extended Memory Server Architecture memory for memory based librefs. When used on startup,
applies to the WORK library.
MEMMAXSZ=2147483648
Maximum amount of memory allocated to support MEMLIB and MEMCACHE options.
MEMSIZE=0 Specifies the limit on the total amount of memory to be used by the SAS System



I hope this displays properly for you. pre and '/pre' don't seem to be working consistently for me. Message was edited by: Chuck

[Cynthia_sas]

Hi, Chuck:
It's not <pre> or </pre> on the forum site. You have to use [ instead of < and ] instead of > ... as in:

[pre]
[ pre ]
[ /pre ]

[/pre]

(I had to add spaces so you could see the characters -- take those out.)
cynthia

[deleted_user]

Oh gee, thanks. No wonder.
Details, details, details.

Ok, I edited the above message multiple times, and '/pre' is still not working correctly for me. When I put in the correct thing, what is displayed is '[' '/p' ']' and all the text after it disappears (is not displayed).

Also, I found that the '=' I had prior to the '[' 'pre' ']' was buggering up that "command". Message was edited by: Chuck

[deleted_user]

I've spoken to sas support and I think they must have a tough job too. But it would be great if there were a knowledge base that shared the work plans and application architectures of successful set ups, so we could leverage on each other's mistakes and lessons. The migration checklist was great, but its hard to tell if something has been amiss (e.g. does SAS EG and Base require tcp tuning done, and how different nodes communicate when its wintel vs unix).

I am rather perturbed that sas support told me that -memsize should be set ard 512M, while some documents suggest it should be 0. So did SAS dump that old formula for NT a long time back ( the one where -memsize approximates to (phys mem - 512) divided by total number of concurrent sessions

[Doc_Duke]

Josh,

In "SAS for Dummies", Chris was referring to using the "Local Computer" vs "SAS server" to get the data. If you are bringing in data from the "local computer", EGuide has to read the data and then send it off to the SAS server for processing (it may be read 2 or more times to accomplish one task and may have to traverse a small pipe). If you use the "SAS Server", then 1) the data are already in a SAS dataset (or linked to a database) and 2) the "SAS Server" accesses the data directly, without any EGuide client involvement.

Depending on what you are trying to do and your hardware environment, one method could be superior to another or vice-versa. There is no universally applicable answer to your question.

Doc