Setting MEMSIZE option in SAS Server

Accepted Solution Solved
Reply
Contributor
Posts: 39
Accepted Solution

Setting MEMSIZE option in SAS Server

Hi All,

I would be grateful if you could please advise how the memsize option can be changed during signon to a SAS Server in SAS 9.3. I have tried differing ways and keep getting the error message that MEMSIZE needs to be specified at startup.

Many thanks

G


Accepted Solutions
Solution
‎05-12-2014 06:51 PM
Contributor
Posts: 39

Re: Setting MEMSIZE option in SAS Server

Thanks for all the responses.  The reason for my question was that the SAS Admin locked down our sessions to a MEMSIZE of 2GB.  However, we required 8GB MEMSIZE as we have a datalayer that is created which utilises Hash Tables.  The process keeps falling over since the SAS Admin made changes to the SAS Server.  We are currently negotiating with the SAS Admin to resolve this issue.   I have also managed to find a way of bypassing the problem by RSUBMIT an x command to call the SAS program and code.

e.g.

x '/apps/sas/9.3/sas -memsize 8192M /data/home/user1234/hash_tables.sas';

Many thanks

G

View solution in original post


All Replies
Super User
Posts: 5,260

Re: Setting MEMSIZE option in SAS Server

As stated in the online doc:

SAS(R) 9.4 Companion for Windows, Third Edition

you need to assign this at startup. In SAS/CONNECT, that means that this has to be defined either in the server config file, autoexec-file or at the SAS startup command line.

Exactly how thsi is done depends on your server SAS/CONNECT configuration, server OS etc.

Contact you SAS server admin to get help on this issue.

Data never sleeps
Super User
Posts: 6,982

Re: Setting MEMSIZE option in SAS Server

If you use SAS/CONNECT, the remote SAS invocation can be found in the login script (eg tcpunix.scr). Add the -memsize option there, or put it in the sasv9.cfg file.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Valued Guide
Posts: 3,208

Re: Setting MEMSIZE option in SAS Server

Please clarify your system/approach as you can use SAS/connect but also Eguide with a Workspaceserver (WS SASapp).

The memsize at a server is, when done pro active, set according the machine capacity and expectation running processes some of them SAS parallel jobs.

When you are getting an "out of memory" message it could also be an indication you are doing something that is exhausting the memory. There will also somewhere limits.

Can you explain when this issue is happening? (code /data/ proc)     

---->-- ja karman --<-----
Solution
‎05-12-2014 06:51 PM
Contributor
Posts: 39

Re: Setting MEMSIZE option in SAS Server

Thanks for all the responses.  The reason for my question was that the SAS Admin locked down our sessions to a MEMSIZE of 2GB.  However, we required 8GB MEMSIZE as we have a datalayer that is created which utilises Hash Tables.  The process keeps falling over since the SAS Admin made changes to the SAS Server.  We are currently negotiating with the SAS Admin to resolve this issue.   I have also managed to find a way of bypassing the problem by RSUBMIT an x command to call the SAS program and code.

e.g.

x '/apps/sas/9.3/sas -memsize 8192M /data/home/user1234/hash_tables.sas';

Many thanks

G

Super User
Posts: 6,982

Re: Setting MEMSIZE option in SAS Server

Reducing MEMSIZE globally is not such a bad idea. At least in our case (AIX), the OS is very intelligent at handling memory and caching, so it is better to let it do that by reducing the SAS memsize, especially if you will have lots of concurrent SAS jobs. This is recommended practice by papers from IBM and SAS.

Our default memsize is 192M(!) for the workspace server and 128M for batch, only some PROC OLAP batch jobs run with ulimit -d unlimited and larger MEMSIZE's.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Valued Guide
Posts: 3,208

Re: Setting MEMSIZE option in SAS Server

Kurt, "gra_in_aus" is proceeding and gone. your reply is too nice to let it in that way.

Trashing is the situation to avoid why you are set those limits. Ho many of us are still knowing that.

http://en.wikipedia.org/wiki/Thrashing_(computer_science)

By setting the upper limit on the memsize in SAS you are setting a upper limit for the process. It is a upper limit as when memory is not needed it will not be used.   

Having well behaved tested validated programs in a operational environment you are not needing a very low limit. The behavior is predictable (like OLTP processing).

Having those analysts with their once runs often being unpredictable in resource-usage it is a different approach. Going for Mining and predictive analytics / forecasting quite that is different IT usage.
Some SAS proc's there are requiring rather high memory settings (above 2Gb) to run acceptable. The IBM papers are giving the uncertainty on this aspects.

AIX that is probably a P7. I guess specifications like 1Tb (or even more) of internal memory and 64 logical processors. As there is a dogma that user processing is not very demanding they are putting a lot of virtual machines machines on that hardware. Each possible having very little processing capacity. Than as lack of capacity we are gong to put them in a grid to get more capacity.
Within RedHat Suse (IBM) there is an option using load segregation cgroup. cgroups - Wikipedia, the free encyclopedia I am missing that at AIX as WLM (WorkLoadlMangement) is offering to little.

That cgroups concept should open a lot of opportunities.

ulimit is at OS level at AIX with SAS is confusing for most of us. You can set that at OS level and it will work with SSH shell usage login. 

When you are using SAS with WS's en SP's the ulimit is not coming for the users profile but from  the spawner (object server). I once did have problem with that and needed to do the research for the forking (sasauth saselsrv) at their concepts. Recently found the sas notes like 38040 - Setting umask and ulimit values for SAS® sessions on UNIX and Linux and 50345 - Changing the current working directory for the SAS® Workspace Server.

For setting memory constraints the Mainframe is most annoying. It is called regionsize but same concept as memsize SAS.
I think most of them still or until recent using a 4Mb setting. That is a faked limit as of the move 24-bit (16Mb) to 32-bit (2Gb) addressing of the z/OS in 90's.
Everyone specifying below 16Mb is getting an additional 32 Mb, above 16Mb you are getting ca 8Mb additional. The sizing between 8Mb-16MB and 2Gb-4Gb are reserved areas.
The java usage is requiring and additional 64Mb in the user area. Than you find yourself that those region-settings are maintained by security-admins declaring everything as dangerous.      

---->-- ja karman --<-----
Super User
Posts: 6,982

Re: Setting MEMSIZE option in SAS Server

IIRC, the main reason for limiting the memsize was the following:

SAS uses available memory to cache the tables. This may lead to the following situation:

SAS requests reads to a file, AIX reads that into its "persistent" (file cache) memory (does a usually very intelligent preload) and lets the process read from there. If SAS is then able to cache on its own, you waste memory by having the same data in memory twice. If several users read the same file, the effect becomes worse, and on top you may run into a situation where file data is swapped out to paging space, completely countering the basic idea of reducing mass storage I/O.

That's why they had the conclusion of letting AIX handle the file caching and keeping SAS to the required minimum (requirements may vary broadly).

According to that, I started with 128M per process and only increased to 192 that when I ran into out of memory conditions when migrating to SAS 9.2; and I set increased limits for certain processes.

Bottom line: I am still serving > 100 EG users with a 2-CPU P520 with 32G of RAM.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Valued Guide
Posts: 3,208

Re: Setting MEMSIZE option in SAS Server

So it was not a technical question, but one because lack on cooperation with your Platform Admin.

The X-cmd usage is often also closed as result of default settings, it has been granted. You are not totally abandoned at all.

When this one would have also locked you could try to use "mp connect" it is having the same effect as X-cmd.

SAS/CONNECT(R) 9.3 User's Guide (mp connect Example 5)

---->-- ja karman --<-----
Valued Guide
Posts: 3,208

Re: Setting MEMSIZE option in SAS Server

Kurt you are doing a nice job having that many users running well. It is a typical workload with not too big datasets (front-end). I assume the datasets being used are limited in size (<2Gb).

With all things you did mention, it would become time to renew some of those parts  

You did not mention the Web-tier and the metadataserver.
The metadataserver is set up to be in memory and needing good response it should not get disturbed by other memory intensive processes.

The webtier can be very memoryintensive also. Getting a system just up you are needing a 12Gb at least. A good running system is needing a lot more.

That 32GB is fine when all is combined as a single server.

Suppose you are getting users that are wanting to process files of 20Gb and much bigger. (backend).

In those cases the system cache will become useless and just causing overhead. Reading your file it won't fit into the memory/cache. Read ahead still working but the cach will flush all content. In those cases you can think on cio/dio like: http://support.sas.com/resources/papers/proceedings09/327-2009.pdf avoiding the system cache and use SAS caching.
The goal is letting other processes on the machines benefit still of the system cache. With 9.2 it can be coded in the SAs programs SAS(R) 9.2 Companion for UNIX Environments.

The SASfile dataset is another to move time form IO to memory just as goal for descreasing responsetimes.


This kind of settings should only be done with a good cooperative technical IT-staff.

Going into the IO as bottleneck there are always trade off's to memory. The 8 byte boundary SAS(R) 9.3 Language Reference: Concepts, Second Edition and a possible 64Kb (system mounts, sas buffer setting) SAS(R) 9.4 Companion for UNIX Environments, Third Edition. Would you run 9.3 last edition, it has been implemented in that version.       

The mentioned Cio/Dio settings (no system cache) is very common with RDBMS. A RDBMS is designed to do IO in an optimal way far beyond what a OS can do. No sequential IO wit an OLTP.   

---->-- ja karman --<-----
🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 9 replies
  • 3725 views
  • 3 likes
  • 4 in conversation