Alerts in SAS Environment Manager

[UlfBorjesson]

I am currently setting up EVM 2.5 on a Win 2008R2 platform running SAS 9.4 M3 in a four server cluster. After partially enable APM the SAS EVM dashboard is reporting alerts about "SASWork Disk Use %>70". It seems APM can not compleatey be enabled since it gives me an error on the kit server, see below.

D:\SAS\Config\Lev1\Web\SASEnvironmentManager\emi-framework\bin>validate.bat --le
vel 2
Service Architecture Validation - Mon Nov 16 11:07:37 EST 2015
Starting level 1 validation...
Level 1 validation completed with 0 error(s)
Starting extended level 2 validation (this could take a while)...

Service Architecture validation completed with 1 error(s)
Errors:
        1) The SAS Environment Manager Server is unable to process additional ta
sks at this time. This may be due to high change request volume. If the server h
as recently been (re)started, it may be synchronizing with its Agents. This proc
ess could take up to an hour.
Please try again at a later time to validate your environment.

This error accoring to EVM is on the metadata server but if one opens MS resource manger and look on disk acitivity disk que is very low .05 and disk activity is low as well.

Has anyone else come across "SASWork Disk Use %>70" alert in EVM and if so how did you reslove it?

All four servers in this cluser are virtual servers on a VMware platform.

Thanks for any input on the matter, Ulf

[BrunoMueller]

Hi

I would say this metric, "SASWork Disk Use %>70", is how full the file system is. So this means the disk where SASWORK is, is more than 70% full. 

Also have a look at this blog entry http://blogs.sas.com/content/sgf/2015/08/19/monitor-your-saswork-directory-from-sas-environment-mana...

bruno

[DeanHatch]

Right now I am receiving the same "SASWork > 70%" alert and the 2.0TB disk has only 18GB (<1%) used on it (according to df command) and the alert is telling me that the dis is 82% full.

That's not useful.

[JuanS_OCS]

Hello @DeanHatch,

if you have any doubts, you can always go to the Inventory tab of the monitored resource by the Alert, and check its configuration. '

Probably the SASWORK defined on SAS foundation it is a different one than the one configured on your SASApp level. Therefore, the disk volumen might differ as well.

It is often the case that the "blame" is not on a "bug" but in our own configurations or our own misunderstanding of the configuration.

Anyway, if you have still questions after that check, you can always come back to us, or ask the question in your SAS Technical Support team.

[DeanHatch]

Juan,

I definitely agree that "It is often the case that the "blame" is not on a "bug" but in our own configurations or our own misunderstanding of the configuration."  However, the Alert is pretty specific about the location of the SAS Work directory and the location of the Alert agrees with the location of the Indicators, which indicates 0% used.

Also, the page for the Resource tells me that no Alerts have been created, which is odd.

Also, this behavior started yesterday when I enabled the SAS Environment Manager Service Architecture for Extended Monitoring.  I have several other nearly identical systems on which I have not enabled the Service Architecture for Extended Monitoring and non of those report such an Alert.  I think in this case it might be called Over-Extended Monitoring, 😉

[DeanHatch]

Recent Alerts

	Date / Time	Alert Name	Resource Name	Fixed	Ack
	07/06/2017 09:45 AM	SASWork Disk Use % > 70	steps2-grid9-2.dev.econ.census…	No
	07/06/2017 09:40 AM	SASWork Disk Use % > 70	steps2-grid9-2.dev.econ.census…	No
	07/06/2017 09:35 AM	SASWork Disk Use % > 70	steps2-grid9-2.dev.econ.census…	No

• Total KBytes Free
  (SAS Home Directory 9.4)	LOW:1.9 TB	AVG:1.9 TB	PEAK:1.9 TB

  

• Use Percent
  (SAS Home Directory 9.4)	LOW:1.0%	AVG:1.0%	PEAK:1.0%

  

[JuanS_OCS]

Hello again @DeanHatch,

no, this is not the definition, either the inventory. What I mean:

- Go to your Resources menu

- Select Services

- Search for MACHINE SAS Home Directory 9.4 SAS work directory

- Select Inventory

- Scroll down and click on the last Edit button

On the directory text box, you will see the directory/logical volume it is actually being monitored by this alert.

[DeanHatch]

You are right, Juan.

The Alert and the Indicator ARE referring to two different locations.

Thanks for the clarification.  I will configure the Alert to Monitor the correct location.

[JuanS_OCS]

No problem, you are welcome.

I usually try to over-understand anything. When I believe to understand something, I try to dive a bit deeper.

FYI, what happened here is:

When you enabled EMI (the extended monitoring), is when all of those alerts where created, and meny things more happened on the background, depending on the components (APM,ACM, Kits, VALoad) you enabled and activated. So this part is expected, and that is why in your other systems you don;t receive the alerts, because they don't exist by default, unless EMI is enabled.

When you enable EMI, a lot of SAS code runs, and feeds the data to the EM agent. This SAS code takes the _SASWORK directory that is defined on your SASFoundation directory. But what people normally calls SASWORK is the same parameter that is normally overwritten at the SASApp level. And here is where the confussion comes from sometimes.

By default, SASWORK on SAS foundation is pointing to /tmp (or the user folder on windows). Therefore SAS recommends to change this configuration to a separated drive/vlolume, with high I/O. But,as said, many people change this config on the SASApp level, and they forget about SAS Foundation, which is a mistake from my point of view, because many calls to SAS maight not be using the configuration of a SASApp server.

As conclussion, my advise is to change this configuration (SASWORK path in the SAS Foundation. And just to skip you from the "pain" of deleting the PWT8155 SAS Home Directory 9.4 SAS work directory resource and to re-define the alerts, it is better if you change the "directory" where we mentioned on the previous post.

[DeanHatch]

"So this part is expected"

You nailed THAT one on the head. I have been using SAS for almost 31 years now and it never ceases to amaze me how counter-intuitive or random some of the designers/developers can be there. The Post-Configuration-Customization-Modifications drive me crazy.

[JuanS_OCS]

🙂 Well, SAS is not easy, historically speaking. However, I feel positive, I see a great improvement since SAS 9.1.3 on post-configuration activities (at that point, every single action was a challenging journey), specially now on SAS 9.4. The more mature the product is, the easier it becomes.

And I hope to see even greater progress with Viya, the expectations are elevated. I think Ansible will help a lot, as explained by @ErwanGranger here https://communities.sas.com/t5/SAS-Communities-Library/Desired-State-is-a-Destination-not-a-Journey/...

I am curious!

[DeanHatch]

Also, couldn't the Monitor and the Indicator use the same location? For me, at least, that would be less confusing than a green check mark being accompanied by an Alert.

[JuanS_OCS]

Yes, they should. But perhaps I am not understanding well here what is a Monitor, and what is an Indicator.

For Hyperic there are Resources (I guess what you call Monitor), and over Resources, Alerts are created. And I have no clue what might be an Indicator here, sorry.