LASR memory issue not going away

[woo]

Hello Friends, 

we have dataload job running (in batch mode) on linux (physical machine - sas 9.4 m5) which has enough memory (around 80 gb) 

we start sas lasr analytics server with command line (script which bring up port 10121) - server starting fine 

when we run a dataload job, it came up with below error. 

data lib1.dataset_def;
set lib2.dataset_abc;
run;

ERROR: Insufficient resources to perform the operation.
ERROR: The SAS System stopped processing this step because of insufficient memory.

we tried all possible way like below to get rid of this error but it simply not going away

- cleaned up LASR library (lib1) - everytime running job with no table in it

- included (-MEMSIZE MAX) for batch server (.../Lev1/SASApp/BatchServer/sasbatch_usermods.sh)

  then when we run test from there, same dir 

 ./sasbatch.sh -nodms

it shows it is running with all physical memory but when i go to home directory and run test batch job (sas test.sas) (proc options group=memory; run;) -> it shows test batch job running with only 2 gb memory. 

i think this is the causing issue for our dataload job (which is running in batch mode). since my test job is running with only 2gb mem, dataload job (which runs in batch mode as well) also picking up only 2 gb mem and errors out.

what way i can use so that when dataload job runs into batch mode, it uses all available memory?

please advise...Thank you. 

[SASKiwi]

Is this the same problem as your earlier post?

https://communities.sas.com/t5/Administration-and-Deployment/MRM-dataload-job-memory-issue/m-p/49053...

If so you should reference it to avoid starting again from the beginning.

I'm going to assume it is the same problem, and that the evidence pointed to you not having enough available VA LASR memory to load your table. 

Check your usage of LASR memory by going into the Administrator option of SAS VA Hub. Here is a screenshot of what you should see. Please confirm if your total Tables Memory is near the limit of your LASR memory or not.

If it is not lack of LASR memory that is the problem please rerun your batch job with the FULLSTIMER option and post your log.

[woo]

Thanks SASKiwi. 

its a same problem but i was running job on linux VM before and now on physical server. Sorry but i didn't want to add/append in old post. 

application va lasr has like ~1400 mb + other two ( public lasr and lasr analytic server) have zero memory from lasr admin tool. job just run for few minutes and then run into memory error right after loading 1 or two tables. 

data lasrlib1.testb;

set lasrlib2.testa;

run;

NOTE: DATA statement used (Total process time):

      real time                 4.12 seconds

      user cpu time         0.80 seconds 

      system cpu time     0.51 seconds 

      memory                  6435.12k

      OS Memory            25696.00k

      Step Count                                 530 Switch Count     54

      Page Faults                                0

      Page Reclaims                           2875

      Page Swaps                               0

      Voluntay Context Switches         625

      Involuntay Context Switches       54

      Block Input Operations                0

      Block Output Operations             0 

[SASKiwi]

Well, again your DATA step stats confirm that it is not your SAS session that is running out of memory. Which LASR library does lasrlib1 point to? What happens if you try loading to a different LASR library like the PUBLIC one?

[JuanS_OCS]

Hello @woo, @SASKiwi,

regarding LASR, there are 2 main points to consider:

- how is the sas.exe session for LASR started.

- how is the sas.exe session that runs the code, started.

Those 2 are different sas processes, and they probably have different configurations, as the memory settings, and limitations.

Once you identify the process for each, you can check memory settings, with "proc options group=memory;run;". The FULLSTIMER is a good idea as well, for this test.

As said, the first thig to do is to identify how the sas session starts. The one for the user, is kind of clear: start a Workspace server or batchServer session, for interactice and bach processes. Or just batchServer if it is a data query that was scheduled.

Then, the remaining question is how that specific LASR server was started.

- If it a server started with Autoload, then it would be a basic SAS Foundation process (see comand line below), but the Autoload.cfg and Autoload_usermods.cfg can override the config settings if needed. For adding my proposed code, you can temporary add it to Autoload.sas.

call "%SAS_COMMAND%" -sysin %FILENAME% -config %CFG_FILE% -log %LOG_FILE% -print %LST_FILE% -batch -noterminal -nologo -logparm "rollover=session"

- If LASR was started manually, with the SAS VA Administrator GUI, then it would be a Workspace Server session, and you would like to check the settings in the sasv9_usermods.cfg of your Workspace Server assigned to SAS VA. To test, just launch a Workspace Server session (EG, DI or just the WorkspaceServer.sh|bat from your server itself.

- LASR can also be started with a custom script, please check it how it istarts, from the script itself.

A last bit: perhaps you would like to consider as well to enable logging for the LASR process itself. Please follow these 2 documents to get more info about your LASR processes.

https://www.wku.edu/instres/documents/vincent_analyzing_lasr_server_logs_sasva.pdf

http://support.sas.com/documentation/cdl/en/inmsref/67213/HTML/default/viewer.htm#n0ki4nsz4f24ptn1x7...

Hope it helps.

Kind regards,

Juan

[nhvdwalt]

Hi @woo

Please could you run the below code just before or after your data step and post the results.

proc options group=memory value;
run;

[woo]

Thank you so much all for your time

here is some more details.  

lasrlib1 - is a lasr library with

LIBNAME mylasr SASIOLA TAG=xyz PORT=10121 HOST="servername.com" SIGNER="http://servername.com:7980/SASLASRAuthorization" ;

- how is the sas.exe session for LASR started.

i am running a custom script to start all lasr servers (script like below)

libname MYLASR SASIOLA
startserver host="servername.com" port=10121
signer="http://servername.com:7980/SASLASRAuthorization";

proc vasmp;
serverwait port=10121;
quit;

- how is the sas.exe session that runs the code, started.
batch mode
sas dataloadjon.sas

- AutoLoad_usermods.cfg and AutoLoad.cfg are not configured with any options in it (like -MEMSIZE or so)

----

before running job i updated sasv9_local from sashome/foundation from compute server and now it is showing session using all physical ram (before it was showing only ~2 gb)

/*TRACE BEFORE DATA STEP*/

proc options group=memory value;
run;

SAS (r) Proprietary Software Release 9.4 TS1M5

Group=MEMORY
Option Value Information For SAS Option SORTSIZE
Value: 1073741824
Scope: SAS Session
How option value set: Config File
Config file name:
/path-to/SASFoundation/9.4/sasv9.cfg

Option Value Information For SAS Option SUMSIZE
Value: 0
Scope: Default
How option value set: Shipped Default

Option Value Information For SAS Option MAXMEMQUERY
Value: 268435456
Scope: SAS Session
How option value set: Shipped Default

Option Value Information For SAS Option LOADMEMSIZE
Value: 0
Scope: Default
How option value set: Shipped Default

Option Value Information For SAS Option MEMSIZE
Value: 126833859840
Scope: SAS Session
How option value set: Config File
Config file name:
/path-to/SASFoundation/9.4/sasv9_local.cfg

Option Value Information For SAS Option REALMEMSIZE
Value: 0
Scope: Default
How option value set: Shipped Default

NOTE: PROCEDURE OPTIONS used (Total process time):
real time 0.00 seconds
user cpu time 0.00 seconds
system cpu time 0.00 seconds
memory 21.71k
OS Memory 24232.00k
Timestamp 10/04/2018 08:59:20 AM
Step Count 631 Switch Count 0
Page Faults 0
Page Reclaims 2
Page Swaps 0
Voluntary Context Switches 0
Involuntary Context Switches 0
Block Input Operations 0

/*JOB CODE*/

data lasrlib1.dataset_abc;
set baselib1.dataset_xyz;
run;

ERROR: Insufficient resources to perform the operation.
ERROR: The SAS System stopped processing this step because of insufficient memory.
NOTE: There were 1585 observations read from the data set xxx.
WARNING: The data set xxx may be incomplete. When this step was stopped there were 1585 observations and 259
variables.
NOTE: DATA statement used (Total process time):
real time 3.24 seconds
user cpu time 0.62 seconds
system cpu time 0.39 seconds
memory 8112.00k
OS Memory 31108.00k
Step Count 632 Switch Count 54
Page Faults 0
Page Reclaims 2674
Page Swaps 0
Voluntary Context Switches 562
Involuntary Context Switches 5
Block Input Operations 0

/*TRACE AFTER DATA STEP*/

proc options group=memory value;
run;

NOTE: PROCEDURE OPTIONS used (Total process time):
real time 0.00 seconds
user cpu time 0.00 seconds
system cpu time 0.00 seconds
memory 22.65k
OS Memory 24232.00k
Step Count 633 Switch Count 0
Page Faults 0
Page Reclaims 2
Page Swaps 0
Voluntary Context Switches 0
Involuntary Context Switches 0
Block Input Operations 0
Block Output Operations 0

[JuanS_OCS]

Hello @woo

Could please share the command line that actually calls the .sas file than includes this code?

@woo wrote:

- how is the sas.exe session for LASR started.

i am running a custom script to start all lasr servers (script like below)

libname MYLASR SASIOLA
startserver host="servername.com" port=10121
signer="http://servername.com:7980/SASLASRAuthorization";

proc vasmp;
serverwait port=10121;
quit;

---

Also, for the logs that you included: whas it in your code that loads the data, or in the program that starts LASR (port 10121)?

When you run this, did you monitor your system (top or htop)? You might be filling up your RAM memory, and even the SWAP, then the sas session (that loads your data to LASR, not the one that start LASR) cannot handle more memory.

[woo]

first, 

i have a script (./start_10121.sh) that i run which include sas program (libname + proc vasmp) which brings up port 10121 (app lasr analytics server). 

second, 

execute sas program (dataload job) in batch mode which include load table (where i receive error)

sas dataloadjob.sas 

yes, i put live watch (top) when i am testing this job where i don't see job is using even more than 1 gb of ram. Now, i am running out option that i can try except something can be updated from lasr admin tool or test job using EG. 

[JuanS_OCS]

Could you share the contents of the ./start_10121.sh script?

[woo]

/path-to/SASFoundation/9.4/sas start_10121.sas -log start_10121.log

[JuanS_OCS]

Good, we are getting there.

And that are the MEMSIZE and SORTSIZE set under your foundation folder?

Or, much better, could you include the proc options proposed at the begining or end of your start_10121.sas code, then re-run it to get the output, and share with us?

[woo]

from compute server, 

sasv9 has

-MEMSIZE 2G

- SORTSIZE 1G

sasv9_local has

-MEMSIZE MAX 

[JuanS_OCS]

All right. Here my sugestions:

1. SAS foundation sasv9 or sasv9_local should never-ever have memsize 0 or memsize max. This will anly cause that all your SAS sessions (LASR, EG, STPs, Pooled Workspace, batches ....) in inherit the same configuration unless overwriten. In any case, it is general recommendation by SAS to not overwrite those files. That is why you have sasv9_usermods files. Reasons, quite important: a) After every maintenance, upgrade, hotfix or migration, the file will be overwriten and customizations would be lost. b) and much-much importantly: memsize max or memsize 0 will ensure that a single sas session will be able to allocate all the memory of your server. Can you imagine what can happen with more than one session taking as much memory? The last session will crash, because it lacks memory .... with the message you get.
2. If you want to start LASR with SAS foundation, it is fine, but use a config file of your own (-config parameter)  or you could even re-use (carefully!) one of the existing ones in your SASApp... That one can have MEMSIZE 0 ... although I would like to recommend that you allocate the memory you need and a bit more for growing and operations. E.g: if your data in LASR is 20GB large, you could assign MEMSIZE 32GB as long as you have that memory in your host where LASR starts.