topic Re: LASR Memory Percentage in SAS Visual Analytics

LASR Memory Percentage

rodrigo_pereira — Tue, 04 Sep 2018 15:29:07 GMT

Dear VA administrator experts,

I'm facing a strange behavior in the LASR memory used by SAS VA 7.4.

When I look to the "used Virtual Memory" of the specific LASR servers it looks in line with what I was expecting. For example, it says that it has 191 GB used.

However, when I look to the top right corner (which represents the total memory) the value is clearly exaggerated and I don't understand why. In this case, it shows 300 GB used when it should be around the same 191 GB.

Can you explain this?

Also, for some explorations I'm getting the following error message and the limits should not have been reached.

"A server-side limit on the consumption of memory resources has been reached. These limits can be adjusted by the owner or by the administrator of the LASR Analytic Server."

I would appreciate very much your help.

Regards,

Rodrigo

Re: LASR Memory Percentage

alexal — Wed, 05 Sep 2018 11:59:33 GMT

@rodrigo_pereira,

I'm wondering what you see in TKGrid Monitor -> File -> Show Jobs on Grid? You can start TKGrid Monitor from your TKGrid folder:

/TKGrid/bin/gridmon.sh

Please note that TKGrid Monitor requires X11 server, local or remote.

Re: LASR Memory Percentage

rodrigo_pereira — Thu, 06 Sep 2018 13:38:30 GMT

@alexal thank you for your answer.

I'm still waiting for the installation of the X11 server ....

Anyway, I've looked at the log in the /opt/sas/sasconfig/Lev1/Applications/SASVisualAnalytics/VisualAnalyticsAdministrator/Monitoring/Logs folder and I've noticed that the LASR is using just 2 worker nodes instead of 3 worker nodes (as it should in our distributed installation!).

Maybe this is the reason.

You know what could be happening? I already have tried to restart (both the LASR server and all the services) but didn't work.

Regards,

Rodrigo

Re: LASR Memory Percentage

alexal — Thu, 06 Sep 2018 13:42:45 GMT

@rodrigo_pereira,

If you do not have access to X11, you can run this SAS program in SAS Studio or EG. We will be able to see exact memory utilization as well as the number of nodes:

LIBNAME LASRLIB SASIOLA  TAG=VAPUBLIC  PORT=<LASR_PORT> HOST="<FQDN_OF_LASR_HEAD_NODE>"  SIGNER="<MIDDLE_TIER_URL>/SASLASRAuthorization" ;

%let sizecols = InMemorySize UncompressedSize 
                CompressedSize TableAllocatedMemory
                InMemoryMappedSize ChildSMPTableMemory
                VirtualMemory ResidentMemory AllocatedMemory;
%let countcols = NumberRecords UseCount RecordLength ComputedColLength;

data tablemem;
    set lasrlib._T_TABLEMEMORY;
run;

data lasrmem;
	set lasrlib._T_LASRMEMORY;
run;

proc print data=lasrmem;
    title "LASR Server Memory Usage";
    format &sizecols.  sizekmg9.2;
    format &countcols. 8.;
    sum _numeric_;
run;

proc print data=tablemem;
    title "LASR Server Table Memory Usage";
    format &sizecols.  sizekmg9.2;
    format &countcols. 8.;
    sum _numeric_;
run;

Re: LASR Memory Percentage

rodrigo_pereira — Thu, 06 Sep 2018 14:54:39 GMT

@alexal thank you!

Unfortunately it's not returning any data (screenshot bellow)

Re: LASR Memory Percentage

alexal — Thu, 06 Sep 2018 15:10:08 GMT

@rodrigo_pereira,

That means you do not have any tables in the memory for that particular LASR server. What about _T_LASRMEMORY?

Re: LASR Memory Percentage

rodrigo_pereira — Thu, 06 Sep 2018 16:03:57 GMT

@alexal for _T_LASRMEMORY it's the same - no data!

That's strange as I've tables loaded in memory:

Re: LASR Memory Percentage

alexal — Thu, 06 Sep 2018 17:46:08 GMT

@rodrigo_pereira,

That is weird. Are you sure that you did specify the correct port in my program? Also, may I review the "Last Action Log" from your LASR server?

Re: LASR Memory Percentage

rodrigo_pereira — Fri, 07 Sep 2018 09:37:37 GMT

@alexal there is bellow a part of the "Last Action Log" from the LASR Server.

NOTE: The LASR procedure is executing in the distributed computing environment with 2 worker nodes.
NOTE: The server started on 'sas-va-1....' port 10011.

NOTE: The LASR Analytic Server port '10011' has been assigned to the macro variable "LASRPORT".

NOTE: The PROCEDURE LASR printed page 1.

In fact, I've noticed an error in the "SIGNER" path and now I can see results in _T_TABLEMEMORY and _T_LASRMEMORY.

What I see is that one of the worker nodes is not being used (we should have 3 ...)

That explains the memory problems ...

However, I don't have a clue why this particular node is not being used by the cluster ....

Re: LASR Memory Percentage

alexal — Fri, 07 Sep 2018 11:16:59 GMT

@rodrigo_pereira,

However, I don't have a clue why this particular node is not being used by the cluster ....

Are you sure that /TKGrid/grid.hosts file contains all nodes? Are you sure this file is the same on all of your nodes?

Re: LASR Memory Percentage

rodrigo_pereira — Fri, 07 Sep 2018 13:51:29 GMT

@alexal yes, the the file contains all the nodes.

do you note something wrong in the memory configurations bellow (resource.settings file) ?

we have one data set around 40 GB and four with around 20 GB.

Should we change the TKMPI_MEMSIYE?

# VM limit (in KBytes). Default is unlimited
#export TKMPI_ULIMIT="-v 50000000"

# Location for temporary files.
#export TKOPT_ENV_UTILLOC=/tmp

# Maximum runtime for non-LASR TKGrid jobs.
#export TKMPI_MAXRUNTIME=3600

# UMask for any files created.
#export TKMPI_UMASK=0022

# Nice level for TKGrid jobs. If unset, defaults to 0 for LASR, 5 for non-LASR
#export TKMPI_NICE=5

# Memory allocation limit (in MBytes). Excludes mmapped files. Default is unlimited.
#export TKMPI_MEMSIZE=30000

# Cgroup to associate with TKGrid jobs.
#export TKMPI_CGROUP="cgexec -g cpu:50"

# Resource Manager.
#export TKMPI_YARN_PRIORITY=2
#export TKMPI_YARN_TIMEOUT=3600
#export TKMPI_YARN_CORES=1
#export TKMPI_RESOURCEMANAGER="java -Xmx256m -Xms256m -cp \"`$HADOOP_HOME/bin/hadoop classpath`\" com.sas.grid.provider.yarn.tkgrid.JobLauncher --masterMem 2000 --javaMem 500 --hostlist \$TKMPI_YARN_HOSTS --cores \$TKMPI_YARN_CORES --memory \$TKMPI_MEMSIZE --priority \$TKMPI_YARN_PRIORITY --timeout \$TKMPI_YARN_TIMEOUT --jobname $TKMPI_APPNAME"

# if [ "$USER" = "lasradm" ]; then
# Custom settings for any process running under the lasradm account.
# export TKMPI_ULIMIT="-v 50000000"
# export TKMPI_MEMSIZE=50000
# export TKMPI_CGROUP="cgexec -g cpu:75"
# fi

# if [ "$TKMPI_APPNAME" = "lasr" ]; then
# Custom settings for a lasr process running under any account.
# export TKMPI_ULIMIT="-v 50000000"
# export TKMPI_MEMSIZE=50000
# export TKMPI_CGROUP="cgexec -g cpu:75"

# Allow other users to read server and tables, but not add or term.
# export TKMPI_UMASK=0033

# Allow no access by other users to lasr server.
# export TKMPI_UMASK=0077

# if [ "$TKMPI_INFO" = "LASRLOAD" ]; then
# TKMPI_INFO is an environment variable that will be passed from
# MVA SAS to the grid. It can be used to distinguish a
# proc lasr create from a proc lasr add, by including
# this line before the proc lasr add:
# options set=TKMPI_INFO="LASRLOAD";
# To exclude from YARN resource manager.
# unset TKMPI_RESOURCEMANAGER
# fi

# Use default nice for LASR
# unset TKMPI_NICE
# fi

# if [ "$TKMPI_APPNAME" = "tklogis" ]; then
# Custom settings for a tklogis process running under any account.
# export TKMPI_ULIMIT="-v 25000000"
# export TKMPI_MEMSIZE=25000
# export TKMPI_CGROUP="cgexec -g cpu:25"
# export TKMPI_MAXRUNTIME=7200
# fi

Re: LASR Memory Percentage

alexal — Fri, 07 Sep 2018 17:38:06 GMT

@rodrigo_pereira,

do you note something wrong in the memory configurations bellow (resource.settings file) ?

I do not see anything wrong. Everything is disabled (by default). Let do this from Linux console:

export GRIDHOST="<LASR_HEAD_NODE_FQDN>"
export GRIDINSTALLLOC=/<PATH_TO>/TKGrid/
export TKPATH=/<PATH_TO>/TKGrid/lib:/<PATH_TO>/TKGrid/bin
export GRIDRSHCOMMAND=/<PATH_TO>/TKGrid/bin/ssh.sh
/<PATH_TO>/TKGrid/bin/checknodes /<PATH_TO>/TKGrid/grid.hosts
/<PATH_TO>/TKGrid/bin/tkgridmon -lasrshow
/<PATH_TO>/TKGrid/bin/tkgridperf
/<PATH_TO>/TKGrid/bin/tkgridmon

Re: LASR Memory Percentage

rodrigo_pereira — Mon, 10 Sep 2018 07:56:22 GMT

@alexalhere is the result.

apparently the node sas-va-3 is in the cluster, but the memory isn't being utilized. I don't understand why ...

Re: LASR Memory Percentage

alexal — Mon, 10 Sep 2018 12:50:41 GMT

@rodrigo_pereira,

Are you sure that you using correct TKGrid folder when starting the LASR server? If so, please open a track and give me the number.

Re: LASR Memory Percentage

rodrigo_pereira — Tue, 11 Sep 2018 07:17:25 GMT

@alexal SAM-I1809 046

Re: LASR Memory Percentage

alexal — Tue, 11 Sep 2018 11:06:31 GMT

@rodrigo_pereira,

That doesn't seem like SAS Technical Support Tracking number.

Re: LASR Memory Percentage

rodrigo_pereira — Tue, 11 Sep 2018 16:07:49 GMT

@alexal it's the incident number to SAS Appplication Management.

Anyway, it seams an infrastructure issue according to the feedback of the support.

Re: LASR Memory Percentage

JuanS_OCS — Wed, 12 Sep 2018 10:43:10 GMT

Hello @rodrigo_pereira,

the output is strange. Yes, I would definetely follow @alexal and SAS Tech Support directions.

Meanwhile, I would like to add a couple of bits, case it might help:

1- To understand and query LASR, I strongly suggest a read to this link: http://support.sas.com/documentation/cdl/en/inmsref/68736/HTML/default/viewer.htm#p050lknh5xepngn1s2kaa5nijrer.htm

2- Additionally to the SAS code, there are 2 alternatives to query LASR:

a- Usage Note 58922: Using the LASRInfo.jar utility to see information about your SAS® LASR™ Analytic Server http://support.sas.com/kb/58/922.html

b- In the SAS VA Administrator web app, and the LASR Tables tab, you can show additional collums. If you show all of them (related to memory) you would see something similar as proposed by @alexal , and as proposed by myself above.

Let me explain to you: The whole point of this exercise is to to have a report that will show the memory loaded by each LASR table. This is important, because the way that LASR behaves is quite clever, but often info can be misleading if you cannot understand.

The info in LASR server tab and the basic info in LASR tables tab, only show up totals, of each table and total of tables. Regarding it sizes in memory.

But "memory" is actually a mix of several kinds of memory: RAM, disk, cache... What you see in the basic information of those tabs, is only the memory as totals of those above. However each table will be loaded in RAM or in disk or cache, depending of statistics and algorithms regarding their actual usage. More usage of data in a table, the more data of that table will be loaded in RAM. The less usage, only a bit in RAM, the rest in disk.

With the report above, you will be able to see how much is actually in RAM, per table, and how much in disk (aka "not really loaded").

To close this explanation, the bar in the top right, do not calculate the basic data of LASR tables and LASR servers, but with all of this additional information. It is a percentage of the REAL limit of your LASR servers, of the total of your memory (RAM + disk/virtual + cache).

To give a quick example:

Imagine you have 5 TB of data loaded. And 5 TB of RAM memory.

So the basic data will show LASR server is full.

Now 2 extreme situations. If your users do not use VA, the bar on top-right can show, e.g, 18%. If your users use VA and the reports and the data at its max, then, the bar can show something close to 100%, which is a sign that your VA environment requires to be resized soon.

Re: LASR Memory Percentage

Filipvdr — Mon, 10 May 2021 08:01:41 GMT

At my client i'm experiencing the following.

Last weeks the %memory usage was climbing towards the limit of 80%. Though, there was not really a lot more data being loaded.

This weekend there was a patch on our linux server, so there was a reboot and our script which loads all data from hadoop to LASR finished and now we are only at 18%.

What is a possible explanation?