topic Re: Modifying alerts in Environment Manager in Administration and Deployment

Modifying alerts in Environment Manager

Nigel_Pain — Mon, 12 Apr 2021 15:07:21 GMT

9.4M7 on Windows 2012R2

I have an alert which has been triggered in EVM for one of our servers, "SASConfig Disk Use % > 95". This is generating an alert every five minutes or so. I'm aware of the issue, and am working to rectify it so I don't need the alerts to keep firing. I tried modifying the alert definition so that it disables until it's been marked as fixed:

but I get an error:

Thanks for any suggestions.

Re: Modifying alerts in Environment Manager

gwootton — Mon, 12 Apr 2021 15:15:37 GMT

Do you see a corresponding error in your Environment Manager's server.log file?

Re: Modifying alerts in Environment Manager

JuanS_OCS — Mon, 12 Apr 2021 15:23:47 GMT

Hello @Nigel_Pain ,

first of all, yes, indeed, as @gwootton indicates, when we get an errors, first thought is to give a look to the logs and the database.

On the other hand, most monitoring tools work though alert vs acknowledgement, and Hyperic, on which SAS EVM is based, is not an exception. You might want to look a the the guide, especially:

Preventing Multiple Alerts for the Same Problem
Disable all Alert Notifications (not recommended)
Setting a Notification Throttle (the best idea)

https://docs.vmware.com/en/vRealize-Hyperic/5.8.4/vrealize-hyperic-584-admin-guide.pdf

This being said, for yous specific problem, I am wondering if you are having a permissions issue either in your file system, in hyperic configuration (and SAS Metadata) or, what could be harder to investigate, in your PSQL database (WIP) for EVM. Hopefully the log should be specific on the matter.

Let us know!

Best regards,

Juan

Re: Modifying alerts in Environment Manager

gwootton — Mon, 12 Apr 2021 16:53:35 GMT

I tested changing this in my environment and encountered a similar failure. To resolve: In addition to checking the box, modify the "greater than" section from "90.0%" to "0.90" and then click Ok.

Re: Modifying alerts in Environment Manager

Nigel_Pain — Mon, 12 Apr 2021 17:05:02 GMT

Thanks @JuanS_OCS and @gwootton for your suggestions.

I would expect to have permissions at all levels: I have an unrestricted SAS account, member of the Superuser role in EVM, and local admins on the server.

The Hyperic guide is just telling me to do what I was trying to!

Presumably the server.log file to check is the one in config\Lev1\Web\SASEnvironmentManager\server-5.8.0-EE\logs? It'll take a bit of searching. A job for tomorrow! Or it maybe easier to tail the log and reproduce the problem.

Re: Modifying alerts in Environment Manager

gwootton — Mon, 12 Apr 2021 17:20:11 GMT

The action you are taking to stop getting multiple alerts is correct (checking the box).
I think the problem is that when you go to edit the alert, Environment Manager is automatically populating the if condition with a unit (%) which the field won't accept. Changing it to a decimal value (i.e. 0.90 instead of 90.0%) allowed me to get past the error.

Re: Modifying alerts in Environment Manager

Nigel_Pain — Tue, 13 Apr 2021 08:47:40 GMT

Thanks Greg. I was quite hopeful when I saw your posting, but unfortunately when I tried this there was no change. I can see some errors in the server.log but they don't mean an awful lot to me:

13-04-2021 09:25:28,580 BST ERROR [tomcat-http--34] [org.hyperic.hq.web.spring.exception.GlobalExceptionController@35] org.hyperic.hq.ui.exception.ParameterNotFoundException: entity ids not found
org.hyperic.hq.ui.exception.ParameterNotFoundException: entity ids not found
 at org.hyperic.hq.ui.spring.util.RequestUtils.getEntityIds(RequestUtils.java:360) ~[sas.evmanager.web.jar:904700.0.0.20200715190000_v940m7]
 at org.hyperic.hq.ui.spring.util.RequestUtils.getEntityId(RequestUtils.java:314) ~[sas.evmanager.web.jar:904700.0.0.20200715190000_v940m7]
 at org.hyperic.hq.ui.spring.action.resource.common.monitor.alerts.config.EditDefinitionConditionsController.eidtConfig(EditDefinitionConditionsController.java:93) ~[sas.evmanager.web.jar:904700.0.0.20200715190000_v940m7]
 at sun.reflect.GeneratedMethodAccessor2403.invoke(Unknown Source) ~[?:?]
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
 at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) ~[spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) ~[spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) ~[spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) ~[spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) [spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) [spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) [spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853) [spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) [servlet-api.jar:?]
 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) [spring-webmvc-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [servlet-api.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106) [spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-websocket.jar:9.0.31.A.RELEASE]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at com.sas.hyperic.security.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:27) [sas.hyperic.security.sso.jar:904700.0.0.20200715190000_v940m7]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at com.sas.hyperic.security.servlet.LocaleServletFilter.doFilter(LocaleServletFilter.java:54) [sas.hyperic.security.sso.jar:904700.0.0.20200715190000_v940m7]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at com.sas.framework.webapp.servlet.SanitizingRequestFilter.onDoFilter(SanitizingRequestFilter.java:101) [sas.framework.webapp.jar:904700.1.0.20200722190000_v940m7]
 at com.sas.framework.webapp.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) [sas.framework.webapp.jar:904700.1.0.20200722190000_v940m7]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.9.RELEASE.jar:3.2.9.RELEASE]
 at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343) [spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260) [spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:230) [spring-orm-3.2.18.RELEASE.jar:3.2.18.RELEASE]
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106) [spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:164) [urlrewritefilter-3.1.0.jar:3.1.0]
 at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141) [urlrewritefilter-3.1.0.jar:3.1.0]
 at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:90) [urlrewritefilter-3.1.0.jar:3.1.0]
 at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:417) [urlrewritefilter-3.1.0.jar:3.1.0]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.hyperic.hq.web.filters.CsrfIgnoreAllGetsFilter.doFilter(CsrfIgnoreAllGetsFilter.java:106) [hq-web-5.8.0-classes.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) [spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106) [spring-web-3.2.18.RELEASE.jar:3.2.18]
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:?]
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:?]
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [catalina.jar:?]
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:?]
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [catalina.jar:?]
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [catalina.jar:?]
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [catalina.jar:?]
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [catalina.jar:?]
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [catalina.jar:?]
 at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-coyote.jar:9.0.31.A.RELEASE]
 at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote.jar:9.0.31.A.RELEASE]
 at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:9.0.31.A.RELEASE]
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639) [tomcat-coyote.jar:9.0.31.A.RELEASE]
 at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:9.0.31.A.RELEASE]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:9.0.31.A.RELEASE]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
13-04-2021 09:25:28,679 BST INFO [tomcat-http--34] [com.sas.framework.webapp.servlet.SanitizingRequestFilter@209] Could not decode the value for POST name: [jsContentBlocks]. Consider adding jsContentBlocks to web.xml allowedPostNames init-param for the SanitizingRequestFilter

Re: Modifying alerts in Environment Manager

gwootton — Tue, 13 Apr 2021 12:25:29 GMT

I think a technical support track may be your best bet here. I'm not sure the "entity IDs not found" error is related to this or not.

One other thing I thought of you might try is making the adjustment to the alert under Manage -> Monitoring Defaults and find the "Edit Alerts" link next to "SAS Config Level Directory 9.4", and try making the same change there.

Re: Modifying alerts in Environment Manager

Nigel_Pain — Mon, 21 Jun 2021 17:07:14 GMT

We finally got this sorted after a lot of investigation by Tech. Support. The analyst was convinced that it was a performance issue but there appeared to be plenty of spare CPU and memory. Then it just started working again (coincidentally, during an online session with Tech. Support). It turned out that our Systems Management had moved the mid-tier VM to a different host, so it was actually host resources that were the problem. Now there's a moral to learn from!

Thanks for your time and effort @gwootton