LASR Memory Percentage

[rodrigo_pereira]

Dear VA administrator experts,

I'm facing a strange behavior in the LASR memory used by SAS VA 7.4.

When I look to the "used Virtual Memory" of the specific LASR servers it looks in line with what I was expecting. For  example, it says that it has 191 GB used.

However, when I look to the top right corner (which represents the total memory) the value is clearly exaggerated and I don't understand why. In this case, it shows 300 GB used when it should be around the same 191 GB.

Can you explain this?

Also, for some explorations I'm getting the following error message and the limits should not have been reached.

"A server-side limit on the consumption of memory resources has been reached. These limits can be adjusted by the owner or by the administrator of the LASR Analytic Server."

I would appreciate very much your help.

Regards,

Rodrigo 

[alexal]

@rodrigo_pereira,

I'm wondering what you see in TKGrid Monitor -> File -> Show Jobs on Grid? You can start TKGrid Monitor from your TKGrid folder:

/TKGrid/bin/gridmon.sh

Please note that TKGrid Monitor requires X11 server, local or remote.

[rodrigo_pereira]

@alexal thank you for your answer.

I'm still waiting for the installation of the X11 server  ....

Anyway, I've looked at the log in the /opt/sas/sasconfig/Lev1/Applications/SASVisualAnalytics/VisualAnalyticsAdministrator/Monitoring/Logs folder and I've noticed that the LASR is using just 2 worker nodes instead of 3 worker nodes (as it should in our distributed installation!).

Maybe this is the reason.

You know what could be happening? I already have tried to restart (both the LASR server and all the services) but didn't work.

Regards,

Rodrigo

[alexal]

@rodrigo_pereira,

If you do not have access to X11, you can run this SAS program in SAS Studio or EG. We will be able to see exact memory utilization as well as the number of nodes:

LIBNAME LASRLIB SASIOLA  TAG=VAPUBLIC  PORT=<LASR_PORT> HOST="<FQDN_OF_LASR_HEAD_NODE>"  SIGNER="<MIDDLE_TIER_URL>/SASLASRAuthorization" ;

%let sizecols = InMemorySize UncompressedSize 
                CompressedSize TableAllocatedMemory
                InMemoryMappedSize ChildSMPTableMemory
                VirtualMemory ResidentMemory AllocatedMemory;
%let countcols = NumberRecords UseCount RecordLength ComputedColLength;

data tablemem;
    set lasrlib._T_TABLEMEMORY;
run;

data lasrmem;
	set lasrlib._T_LASRMEMORY;
run;

proc print data=lasrmem;
    title "LASR Server Memory Usage";
    format &sizecols.  sizekmg9.2;
    format &countcols. 8.;
    sum _numeric_;
run;

proc print data=tablemem;
    title "LASR Server Table Memory Usage";
    format &sizecols.  sizekmg9.2;
    format &countcols. 8.;
    sum _numeric_;
run;

[rodrigo_pereira]

@alexal thank you!

Unfortunately it's not returning any data (screenshot bellow)

[alexal]

@rodrigo_pereira,

That means you do not have any tables in the memory for that particular LASR server. What about _T_LASRMEMORY?

[rodrigo_pereira]

@alexal for  _T_LASRMEMORY it's the same - no data!

That's strange as I've tables loaded in memory: 

 

[alexal]

@rodrigo_pereira,

That is weird. Are you sure that you did specify the correct port in my program? Also, may I review the "Last Action Log" from your LASR server?

[rodrigo_pereira]

@alexal there is bellow a part of the "Last Action Log" from the LASR Server.

NOTE: The LASR procedure is executing in the distributed computing environment with 2 worker nodes.
NOTE: The server started on 'sas-va-1....' port 10011.

NOTE: The LASR Analytic Server port '10011' has been assigned to the macro variable "LASRPORT".

NOTE: The PROCEDURE LASR printed page 1.

In fact, I've noticed an error in the "SIGNER" path and now I can see results in _T_TABLEMEMORY and _T_LASRMEMORY.

What I see is that one of the worker nodes is not being used (we should have 3 ...)

That explains the memory problems ...

However, I don't have a clue why this particular node is not being used by the cluster ....

[alexal]

@rodrigo_pereira,

However, I don't have a clue why this particular node is not being used by the cluster ....

Are you sure that /TKGrid/grid.hosts file contains all nodes? Are you sure this file is the same on all of your nodes?

[rodrigo_pereira]

@alexal yes, the the file contains all the nodes.

do you note something wrong in the memory configurations bellow (resource.settings file) ?

we have one data set around 40 GB and four with around 20 GB.

Should we change the TKMPI_MEMSIYE?

# VM limit (in KBytes). Default is unlimited
#export TKMPI_ULIMIT="-v 50000000"

# Location for temporary files.
#export TKOPT_ENV_UTILLOC=/tmp

# Maximum runtime for non-LASR TKGrid jobs.
#export TKMPI_MAXRUNTIME=3600

# UMask for any files created.
#export TKMPI_UMASK=0022

# Nice level for TKGrid jobs. If unset, defaults to 0 for LASR, 5 for non-LASR
#export TKMPI_NICE=5

# Memory allocation limit (in MBytes). Excludes mmapped files. Default is unlimited.
#export TKMPI_MEMSIZE=30000

# Cgroup to associate with TKGrid jobs.
#export TKMPI_CGROUP="cgexec -g cpu:50"

# Resource Manager.
#export TKMPI_YARN_PRIORITY=2
#export TKMPI_YARN_TIMEOUT=3600
#export TKMPI_YARN_CORES=1
#export TKMPI_RESOURCEMANAGER="java -Xmx256m -Xms256m -cp \"`$HADOOP_HOME/bin/hadoop classpath`\" com.sas.grid.provider.yarn.tkgrid.JobLauncher --masterMem 2000 --javaMem 500 --hostlist \$TKMPI_YARN_HOSTS --cores \$TKMPI_YARN_CORES --memory \$TKMPI_MEMSIZE --priority \$TKMPI_YARN_PRIORITY --timeout \$TKMPI_YARN_TIMEOUT --jobname $TKMPI_APPNAME"

# if [ "$USER" = "lasradm" ]; then
# Custom settings for any process running under the lasradm account.
# export TKMPI_ULIMIT="-v 50000000"
# export TKMPI_MEMSIZE=50000
# export TKMPI_CGROUP="cgexec -g cpu:75"
# fi

# if [ "$TKMPI_APPNAME" = "lasr" ]; then
# Custom settings for a lasr process running under any account.
# export TKMPI_ULIMIT="-v 50000000"
# export TKMPI_MEMSIZE=50000
# export TKMPI_CGROUP="cgexec -g cpu:75"

# Allow other users to read server and tables, but not add or term.
# export TKMPI_UMASK=0033

# Allow no access by other users to lasr server.
# export TKMPI_UMASK=0077

# if [ "$TKMPI_INFO" = "LASRLOAD" ]; then
# TKMPI_INFO is an environment variable that will be passed from
# MVA SAS to the grid. It can be used to distinguish a
# proc lasr create from a proc lasr add, by including
# this line before the proc lasr add:
# options set=TKMPI_INFO="LASRLOAD";
# To exclude from YARN resource manager.
# unset TKMPI_RESOURCEMANAGER
# fi

# Use default nice for LASR
# unset TKMPI_NICE
# fi

# if [ "$TKMPI_APPNAME" = "tklogis" ]; then
# Custom settings for a tklogis process running under any account.
# export TKMPI_ULIMIT="-v 25000000"
# export TKMPI_MEMSIZE=25000
# export TKMPI_CGROUP="cgexec -g cpu:25"
# export TKMPI_MAXRUNTIME=7200
# fi

[alexal]

@rodrigo_pereira,

do you note something wrong in the memory configurations bellow (resource.settings file) ?

I do not see anything wrong. Everything is disabled (by default). Let do this from Linux console:

export GRIDHOST="<LASR_HEAD_NODE_FQDN>"
export GRIDINSTALLLOC=/<PATH_TO>/TKGrid/
export TKPATH=/<PATH_TO>/TKGrid/lib:/<PATH_TO>/TKGrid/bin
export GRIDRSHCOMMAND=/<PATH_TO>/TKGrid/bin/ssh.sh
/<PATH_TO>/TKGrid/bin/checknodes /<PATH_TO>/TKGrid/grid.hosts
/<PATH_TO>/TKGrid/bin/tkgridmon -lasrshow
/<PATH_TO>/TKGrid/bin/tkgridperf
/<PATH_TO>/TKGrid/bin/tkgridmon

[rodrigo_pereira]

@alexalhere is the result.

apparently the node sas-va-3 is in the cluster, but the memory isn't being utilized. I don't understand why ...

[alexal]

@rodrigo_pereira,

Are you sure that you using correct TKGrid folder when starting the LASR server? If so, please open a track and give me the number.

[rodrigo_pereira]

@alexal SAM-I1809 046