SAS Environment Manager monitors and administers many aspects of a SAS 9.4 deployment. EV (as we refer to it internally) is comprised of two different software components: a Server and its Agents. An agent is deployed to each host of a SAS deployment where it gathers information about ongoing operations. The agent relays that information back to the EV server where it can be analyzed and shared with the SAS software's administration personnel.
In this way, SAS Environment Manager is flexible to the many possible topologies of SAS software deployments and enables administration, alerts, auditing, logging, and many other features to help keep the SAS deployment running smoothly.
However, there are some SAS deployment topologies for which EV needs configuration beyond what the standard install process can provide. In particular, SAS installations which require additional configuration of the EV agents are those where the SAS Compute Tier runs across multiple hosts machines which all share a single deployment of the SAS configuration directory. In the past we relied on a separately downloadable utility to help configure the EV agents for this scenario. But with SAS 9.4 M6, we have a new smart agent technology to make it even easier.
This post dives right into some of the deeper aspects of multi-tiered, multi-machine deployments of SAS. Where possible, I'll try to provide relevant background to keep us all on the same happy path.
A multi-tiered SAS deployment is a reference to distributing SAS software components to different machines based on their primary function. We typically see tiers referenced like:
- When talking about the new EV Smart Agent functionality, we're focused exclusively on the SAS Compute Tier.
It comes down to how the SAS software is deployed initially and then accessed for ongoing operations. To understand why the compute tier is different, let's first look at how other SAS software tiers are deployed. Consider this clustered SAS Metadata Server on three host machines:
Select any image to see a larger version.
To deploy the SAS Metadata Server, we must run the SAS Deployment Wizard (SDW) on each host. The SDW lays down physical files for sashome and sasconfig on to each machine. Once the files are in place, then the SDW starts up the SAS software services on that host automatically, including the EV Agents.
The process is pretty similar for a clustered SAS Middle Tier as well:
In each case, every host machine has its own set of sashome and sasconfig files - and each one has its own SAS Environment Manager Agent as well. Also, one (or more) of these hosts is also running the SAS Environment Manager Server component, too.
So if you're keeping count, you'll notice that this approach requires running the SDW five times so far - once on each machine shown in these two illustrations.
The SAS Compute Tier can be treated differently. It's possible that the SAS Compute Tier might not just have one, two, or three hosts - it may have dozens, scores, potentially hundreds of hosts. Instead of running the SDW on every single one, wouldn't it be great if we could just run the SDW once and share that one set of files across all of the compute tier hosts?
TAKE NOTE: The SAS Compute Tier for this post is in reference to the software components which provide Foundation SAS through connectivity provided by either SAS/CONNECT or SAS Integrations Technology (IOM) software. This includes the SAS/CONNECT Server or SAS processes spawned by the SAS/CONNECT Spawner as well as IOM application servers like the SAS Workspace Server, SAS Stored Process Server, SAS Pooled Workspace Server, and SAS OLAP Server. And so, of course, SAS Grid Manager is included in this, too. For this post, the compute tier does not include SAS LASR Analytic Server, SAS Event Stream Processing, anything from SAS Viya, etc.
For the hosts in the SAS Compute Tier, we can choose to share one instance of the SAS configuration directory. And if our SAS deployment is running on Linux (or UNIX) hosts, then we can also share the sashome directory (with all of SAS' binary executables) as well:
In this illustration, notice that the SDW is only run on one host machine of the compute tier. All of the other compute tier hosts will be configured to access the files for sashome and sasconfig through the use of shared filesystem technology. This simplifies the deployment significantly since we only need run the SDW for the compute tier just one time. It also simplifies ongoing administration and maintenance because we can make one change (or apply a hotfix just once) and all of the machines in the compute tier will see it automatically.
Yes, indeed. I'm just getting to it. But first you need to know that the standard EV Agent is already installed, configured, and running on every host where the SDW was executed. In the illustrations above, the SDW has already provided a perfectly good EV agent on:
We don’t change those. Leave them alone. They're fine.
However, notice that we don't have any EV agents running on the compute tier hosts where the SDW was not executed: comp02, comp03, comp04, …, compn.
This is where the new EV Smart Agent comes into play.
This is actually super simple. There's nothing to deploy. As of 9.4 M6, the EV Smart Agent software files are already in place in the sasconfig directory. You just need to use them.
On every SAS Compute Tier host which relies on a shared configuration *and* has not run the SDW:
/[SASCONFIGDIR]/Lev1/Web/SASEnvironmentManager
drwxr-xr-x. 8 sasinst sas 195 Dec 11 11:06 agent-5.8.0-EE drwxr-xr-x. 11 sasinst sas 168 Dec 11 11:05 emi-client drwxr-xr-x. 18 sasinst sas 4096 Dec 11 11:04 emi-framework drwxr-xr-x. 2 sasinst sas 91 Dec 11 11:05 smart-agent
Notice the standard EV agent directory is still there: agent-5.8.0-EE. And remember it's actively being used on the compute tier host machine where the SDW did run.
-rw-r--r--. 1 sasinst sas 2213 Dec 11 11:07 agent.properties -rw-r--r--. 1 sasinst sas 7461 Dec 11 11:05 hq-agent.bat -rwxr-x--x. 1 sasinst sas 6638 Aug 30 00:33 hq-agent.sh -rw-r--r--. 1 sasinst sas 264 Dec 11 11:05 sas.properties
hq-agent.sh
script and specify the start parameter. For it's first-ever execution, the smart agent will automatically create a new directory structure just for this host's files:
[running on comp02] $ ./hq-agent.sh start Cloning a new HQ Agent for host comp02.. Cloned a new HQ Agent for host comp02 under the directory /[SASCONFIGDIR]/Lev1/Web/SASEnvironmentManager/cloned_agents/comp02/agent-5.8.0-EE. Starting HQ Agent...... running (19889). [ Running agent setup ] Should Agent communications to HQ be unidirectional [default=no]: yes What is the HQ server IP address: mid01.site.com Should Agent communications to HQ always be secure [default=yes]: Yes What is the HQ server SSL port [default=7443]: 7443 - Testing secure connection ... Success What is your HQ login [default=hqadmin]: sasevs@saspw What is your HQ password: **Not echoing value** What is the agent IP address [default=10.97.2.78]: comp02 - Received temporary auth token from agent - Registering agent with HQ - HQ gave us the following agent token 1546870783922-4246574249650571876-6661908361341976489 - Informing agent of new HQ server - Validating - Successfully setup agent - Agent using new transport, unidirectional=true, port=7443
/[SASCONFIGDIR]/Lev1/Web/SASEnvironmentManager +-- cloned_agents +-- [HOSTNAME] +-- agent-5.8.0-EE +-- bin +-- bundles +-- conf +-- data +-- log +-- wrapper -- open_source_licenses-SpringSource_tc_Server_Plug-in.txt -- open_source_licenses.txt -- README.txt
/[SASCONFIGDIR]/Lev1/Web/SASEnvironmentManager/smart-agent/hq-agent.sh [ start | stop | restart | status ]
Let's be clear. For every machine where the SDW ran, we already have the standard EV Agent and it works great. Each of those hosts where the SDW ran also has its own sas.servers
utility to start (and stop and status) all of the SAS services, including the default EV Agent. So continue working with those as you always have.
The EV Smart Agent is used on the compute tier only - and only those hosts which rely on the shared SAS configuration directory - and only those which did not run the SDW. So your SAS deployment when its complete will run with the standard EV Agent running on some hosts and the EV Smart Agent on the compute tier hosts which need it.
This is important to remember. Here's why…
On the first-deployed host of the SAS Compute Tier, you'll typically use the sas.servers
utility to operate all of the SAS services together. But if you want to operate just the EV Agent, you can do so by calling the correct hq-agent.sh
script:
$ /[SASCONFIGDIR]/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/bin/hq-agent.sh status HQ Agent is running (PID:4812). Current agent bundle: agent-5.8.0 Server IP address: mid01.site.com Server (SSL) port: 7443 Using new transport; unidirectional=true Agent listen port: 2144
However, on the other hosts of the SAS Compute Tier which are running the EV Smart Agent, the command path is different:
$ /[SASCONFIGDIR]/Lev1/Web/SASEnvironmentManager/smart-agent/hq-agent.sh status HQ Agent is running (PID:1485). Current agent bundle: agent-5.8.0 Server IP address: mid01.site.com Server (SSL) port: 7443 Using new transport; unidirectional=true Agent listen port: 2144
I know. You want to run the EV Smart Agent on the first-deployed compute tier host (where the SDW ran). And technically, you can. It'll work (assuming you stop the standard EV agent first) and report statistics to the EV Server just the same. But it's a bad idea.
It's a bad idea because the sas.servers
utility - which you want to continue using - is not equipped to work with the EV Smart Agent. So you'd end up wrangling with competing executions of the EV agent to no real benefit.
Okay, now I bet you want to run the standard EV Agent on the other hosts of the compute tier, right? And well, you can. It's just that's the old way of doing things. And the EV Smart Agent is the new way.
Unfortunately, no one can be told what the Matrix is. You have to see it for yourself.
Here's an interesting twist to consider: there is no EV Smart Agent.
As I said above, whether you're running the standard EV Agent or the new EV Smart Agent, you're getting the exact same agent-5.8.0 process. It's just that the parameters fed to that agent which tell it how to run are different. The "smarts" are only in the scripting which invokes the agent - specifically to create that dedicated directory structure per host and direct the agent to use it. In terms of what the EV agent reports on and monitors, the standard and smart agents are exactly the same.
With so much power and flexibility in how we deploy SAS software, there's still a lot of ground to cover when discussing the concepts here, not just for SAS Environment Manager, but the various topologies, operational considerations, and so on.
Learn more about saving time and effort with a shared configuration directory.
And when working with a multi-tiered (or multi-machine) deployment of SAS software, coordination of SAS software services as they run across hosts is important. SAS Technical Support provides the SAS_lsm utility to help manage the operations of SAS software on multiple machines.
@RobCollum , thank you very much for sharing. HUGE improvement. I will definetely play with it and test it. Specially usefull for SAS Grid deployments.
Thanks for this! As mentioned, this is very nice for a Grid environment, and was very timely information for completing our new 9.4M6 Grid just today 🙂
Thank you for this Rob... had I not been reading up on the new SAS Workload Orchestrator I would have missed setting this up.
Makes perfect sense!
Renée La Forest
Hi Rob
Great post. But would it be possible to have the cloned agent run on all compute nodes on a grid and then alter the sas.server.mid and sas.server.mid.template script to point to the cloned_agent for stop/start/status
SHOSTNAMEUNX=`hostname`
SASENVMGRAGNT_DIR="$LEVELDIR/Web/SASEnvironmentManager/cloned_agents/$SHOSTNAMEUNX/agent-5.8.0-EE"
if [ "$OSTYPE" = "aix" ];
then
SASENVMGRAGNT_CMD="$SASENVMGRAGNT_DIR/bundles/agent-5.8.0/bin/hq-agent-nowrapper.sh"
else
SASENVMGRAGNT_CMD="$SASENVMGRAGNT_DIR/bin/hq-agent.sh"
fi
I have had this setup run at a customer and haven't run into any problems yet.
Best Regards
Håvard
this is an interesting approach, however, I would like to ask you:
Have you got the chance to check with top, htop or any resource monitoring tool, how many resources (specially CPU) are in use by the cloned agents?
I would like to hear your experiences in this regard, @HåvardPehrson and @RobCollum
Best regards,
Juan
Hi @JuanS_OCS
No. I haven't used any resource monitoring tools to check cpu usage. However, I would expect any significant difference compared to the old setup.
Best Regards
Håvard
Thank you for the thought.
In my experience, the cloned agent is using much more resources. While the normal EVM agent uses 1%-2% as much, I perceive the cloned agent often uses ~20% cpu, which I think is something to pay attention to, specially in grid environments.
Hi @JuanS_OCS
I checked on my server and I don't see any increased cpu usage for the smart agent pid
[smart-agent]$ ./hq-agent.sh status
HQ Agent is running (PID:21125).
top - 11:48:29 up 17 days, 22:44, 2 users, load average: 0,18, 0,36, 0,58
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1,1 us, 0,9 sy, 0,0 ni, 97,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
KiB Mem : 65361320 total, 34909836 free, 24110000 used, 6341484 buff/cache
KiB Swap: 2097148 total, 48 free, 2097100 used. 39747240 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21125 dsas 20 0 136368 1200 640 S 0,0 0,0 0:26.28 wrapper-linux-x
But thanks, that is something to keep in mind 🙂
Best Regards
Håvard
RE: alter the sas.server.mid and sas.server.mid.template script to point to the cloned_agent for stop/start/status
I'm glad that works for you! Personally, I don't recommend manually changing those files. The next time you install a hotfix or maintenance update, those files are fair game to be replaced. I'm sure the dev teams would enjoy hearing your request for that functionality. I'll share with them, but best would be if you made a feature request. SAS Technical Support should be able to help with that.
RE: 20% CPU utilization:
As I wrote above:
...whether you're running the standard EV Agent or the new EV Smart Agent, you're getting the exact same agent-5.8.0 process. It's just that the parameters fed to that agent which tell it how to run are different. The "smarts" are only in the scripting which invokes the agent - specifically to create that dedicated directory structure per host and direct the agent to use it. In terms of what the EV agent reports on and monitors, the standard and smart agents are exactly the same.
If you're experiencing unusual resource consumption, then I suggest opening a track with SAS Technical Support to run down the cause. I wouldn't want a simple monitoring agent to consume so much processing power.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 16. Read more here about why you should contribute and what is in it for you!
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.