Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

[rgreen33]

We are attempting to initialize and enable the Service Architecture in our SAS environment (Distributed with Hadoop - Linux).  I am following this documentation (http://support.sas.com/documentation/cdl/en/evug/69029/HTML/default/viewer.htm#n18wwqg60hkjqon12qo39... ).  We have successfully initialized the SAS Environment Manager Extended Monitoring.  We have successfully Enabled and Initialized the APM ETL.  Now, we are on the last step for Enabling ACM ETL.  When we attempt to run the command "./emi_init.sh --enable ACM", it is hanging at "Starting metricData".

Here's the output from the command:

[sas@sasdr bin]$ ./emi_init.sh --enable ACM

Service Architecture Initialization Utility - Tue Mar 14 08:56:05 EDT 2017

Validating environment...

Validating external connections...

ACM Enabled

        Running ACM ETL...

Service Architecture ACM ETL Process - Tue Mar 14 08:56:19 EDT 2017

Starting dbMirrorData...

Starting inventoryUpdate...

Service Architecture Group Creation - Tue Mar 14 08:56:31 EDT 2017

Creating/updating groups...

Successfully processed 11 groups in 25.346 sec

Starting metricData

And it just hangs here...for hours.

The end of the log file looks like this:

***Log file getMetricData_20170314_085619.sas.log***

NOTE: Appending WORK.HQ_METRIC_DATA to EVDMRAW.HQ_METRIC_DATA.

NOTE: There were 0 observations read from the data set WORK.HQ_METRIC_DATA.

NOTE: 0 observations added.

NOTE: The data set EVDMRAW.HQ_METRIC_DATA has 1 observations and 3 variables.

NOTE: PROCEDURE APPEND used (Total process time):

      real time           0.00 seconds

      user cpu time       0.00 seconds

      system cpu time     0.00 seconds

      memory              565.65k

      OS Memory           18088.00k

      Timestamp           03/14/2017 08:57:06 AM

      Step Count                        18  Switch Count  0

      Page Faults                       0

      Page Reclaims                     55

      Page Swaps                        0

      Voluntary Context Switches        0

      Involuntary Context Switches      0

      Block Input Operations            0

      Block Output Operations           144

^L8                                                          The SAS System                              08:57 Tuesday, March 14, 2017

NOTE: DATA statement used (Total process time):

      real time           0.00 seconds

      user cpu time       0.00 seconds

      system cpu time     0.00 seconds

      memory              806.59k

      OS Memory           18088.00k

      Timestamp           03/14/2017 08:57:06 AM

      Step Count                        19  Switch Count  0

      Page Faults                       0

      Page Reclaims                     221

:$

      Page Reclaims                     5

      Page Swaps                        0

      Voluntary Context Switches        0

      Involuntary Context Switches      0

      Block Input Operations            0

      Block Output Operations           8

NOTE: DATA statement used (Total process time):

      real time           0.00 seconds

      user cpu time       0.00 seconds

      system cpu time     0.00 seconds

      memory              805.68k

      OS Memory           17832.00k

      Timestamp           03/14/2017 08:57:06 AM

      Step Count                        47  Switch Count  0

      Page Faults                       0

      Page Reclaims                     217

      Page Swaps                        0

^L18                                                         The SAS System                              08:57 Tuesday, March 14, 2017

      Voluntary Context Switches        0

      Involuntary Context Switches      0

      Block Input Operations            0

      Block Output Operations           0

NOTE: DATA statement used (Total process time):

      real time           0.01 seconds

      user cpu time       0.01 seconds

      system cpu time     0.01 seconds

      memory              816.28k

      OS Memory           17832.00k

      Timestamp           03/14/2017 08:57:06 AM

      Step Count                        48  Switch Count  51

      Page Faults                       0

      Page Reclaims                     220

      Page Swaps                        0

      Voluntary Context Switches        330

      Involuntary Context Switches      1

      Block Input Operations            0

      Block Output Operations           0

What am I missing?

Thanks,

Ricky

[JuanS_OCS]

Hello @rgreen33,

if you have big amount (or size) of logs from your SAS services, and from your SAS Environment Agent (the data folder), it could be that it is processing all of this data.

Remember: The SAS logs will not write a running data step, until the data step is complete. So, the last data step you could see, is the last completed data step.... but if you go to the related SAS code that is being executed, most probably you will see addtiional data steps or procedures, and the first one will the the one currently running. I can easily imagine that it is a process that is loading data from the logs or your agent metrics into the WIP database (your EVDM library).

You can also check the running sas.exe processes and the activity on your server resources.

Just for fun and giggles, it could be interesting to enable emi_init after a clean/archive of your logs and agent data?

[rgreen33]

@JuanS_OCS,

Thank you for your reply.  We initially kicked the process off around 3:00 PM yesterday.  This morning, it still appeared to be hanging.  Watching the log file and the files under Datamart folders, we are not seeing any activity.  It goes to a certain point and then appears to hang.  We killed the process this morning and kicked it off again...same thing happened. 😞  Watching the service in Environment Manager, it goes from red to green to yellow and then back to red.  Do you know if there is a way to see if it is truly still active?  Anything that I can watch to see the activity?

Thank you,

Ricky

[JuanS_OCS]

Hello @rgreen33,

I have seen similar behaviour on other processes, and on this process trying to access network resources without any success.

http://support.sas.com/documentation/cdl/en/evug/69029/HTML/default/viewer.htm#p0jgzt0ahiohlqn1c9auz...

This link explains the pre-requisites, and some of them speak about the SSL and network shares. Did you have the opportunity to check all of them?

[rgreen33]

@JuanS_OCS,

Thanks again for your help.  After re-reading the prerequisites, I may have found the issue.  Per the prerequisites...

Ensure that your SAS deployment contains a functioning Stored Process server. The server must be defined on the SAS Environment Manager Enablement Kit Server, and it cannot be load balanced across multiple machines.

Looking more closely at my environment, I found that I may have Stored Process Server load balanced across multiple machines.  Here's what my SASApp - Stored Process Server looks like in SAS Management Console:

Could this be my issue?  If so, does that mean that I will NOT be able to run ACM?  What about APM?  It looks as if everything is working fine with APM.  Should I disable it since the SASApp - Stored Process Server is load balanced?

Thanks again for your help,

Ricky

[JuanS_OCS]

Hi there @rgreen33,

no, your Stored Process server is completely normal. it is load balanced (by default, as it should be) and in a single server (it is possible to configure the Stored Process to run on multiple servers). So this one should not be your problem.

From https://support.sas.com/documentation/cdl/en/evug/69029/PDF/default/evug.pdf

Spoiler

Working with ACM ETL
The ACM ETL process extracts and stores metric data that is collected by the SAS
Environment Manager agents into the data mart. Storing the data in the data mart
enables you to access the data for a longer period of time than if it was stored only in
the SAS Environment Manager web application.
ACM data is processed and loaded into the data mart in these steps:
1 Metric data is collected from the SAS Environment Manager agents and sent to the
SAS Environment Manager database.
2 At specified intervals, the ACM ETL process runs. The process copies data from the
database, standardizes the data, and loads the data into the data mart.

So, I still think it is a question related to the agents. A couple of checks I would propose you:

1. If you go to the SAS Environment Manager, to Resources, are all of your Platforms with green check marks?
2. What is the total size in GB  of every /Web/SASEnvironmentManager/agent-XXXXX/data folder of every node on your deployment?
3. What is the size of your ACM tables? http://support.sas.com/documentation/cdl/en/evug/69029/HTML/default/viewer.htm#n13wvzy0q896tsn19cqms...

These tables are located in the directory [levelroot]/Web/SASEnvironmentManager/emi-framework/Datamart/acm.
ACM.AVAILABILITY Table
ACM.FILEMOUNTS Table
ACM.GROUPINVENTORY Table
ACM.EVENTS Table
ACM.HOSTPLATFORMS Table
ACM.HTTPCHECKS Table
ACM.IOMSERVERS Table
ACM.MEASUREINVENTORY Table
ACM.METADATASVRS Table
ACM.NETWORKINTERFACE Table
ACM.RESOURCEINVENTORY Table
ACM.SASAPPSERVERS Table
ACM.SASLOGICALSERVERS Table
ACM.TCSSERVERMGRS Table
ACM.WEBAPPSERVER Table
ACM.WIPDATADB Table

[rgreen33]

@JuanS_OCS,

So, I checked the Environment Manager Resouces...and all are green (2 of them).

I checked on the size of the files in our /Web/SASEnvironmentManager/agent-XXXXX/data folder.  Here are the results:

[sas@sasdr data]$ du -hs /data/sasconfig/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/data

37M     /data/sasconfig/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/data

I also checked the file of our Datamart acm files.  We only have 4 files in that folder:

Does this help?

Thanks,

Ricky

[rgreen33]

@JuanS_OCS,

One thing to add...this is a fresh install.  The install of our servers "completed" and we are just now getting users setup and running.  We are working out some of these "kinks" before we let too many people get on the systems.  One thing to note...not sure if it is related to this or not...but, we do have several ActiveMQ 5.7 Topic services that are currently showing up in Environment manager as "unavailable".  We are just trying to tackle one issue at a time.

Thanks,

Ricky

[JuanS_OCS]

Hello @rgreen33,

I am practically sure that it is related to the agents. In case you have inactive ActiveMQ services, you probably might want to understand what is going on those.

Could you please send a screenshot of your SAS Environment Manager, the Resources page, and the full list of Servers tab?

What errors can you find on the logs of the ActiveMQ?

Also, it might be an idea to start a track with your SAS Technical Support in pararell/

Thank you in advance.

Juan

[chardin]

Juan,

I'm adding some attachments on Ricky's behalf, while he is away. I work with him in supporting the same SAS environment.

One of the screenshots provides a list view on all the EM Resources - Servers. The other shows a list view of all the EM Resources - Unavailable Services as well. The only error I found in the ActiveMQ log file (/data/sasconfig/Lev1/Web/activemq/data/activemq.log) was dated Nov. 28, 2016. The same log contains several warning messages which are more current and shows numerous failed connections. I’ve included Word documents for each of those.

Please note that a SAS Suport ticket/track has been opened on this issue as well. I hope this information helps.

Thank you for the additional assistance.

Carl

[chardin]

Attachment 2 showing the ActiveMQ error...

[chardin]

Attachment 3 showing SAS Environment Manager - Resources - All Servers...

[chardin]

Attachment 4 showing SAS Environment Manager - Resources - Unavailable Services...

[JuanS_OCS]

Hello @chardin,

thank you for taking over @rgreen33 's communications!

What I can see here, is that your ActiveMQ connections (your JMS Broker, the metrics collector, I guess) is starting when there is a same process already started, blocking the port number.... or your firewall is giving the problems. Maybe the services were not stoped properly and there are some roge processes that are blocking your ports. In this case, netstat and ps are our commands, then kill the processes.

Once you do this, and you ensure no SAS processes are started, just try to start al the services on the nodes.

Could you please attach the 4th attachment? Something was wrong when you tried to post it.

PS. Something that might help:

For firewall and network issues:

http://support.sas.com/kb/43/946.html

http://support.sas.com/kb/52/670.html

Solve ActiveMQ errors:

http://support.sas.com/kb/54/066.html

http://support.sas.com/kb/52/315.html

[chardin]

Juan,

Thanks for the URL's. I'll review them next. Here is the missing attachment showing the unavailable services.

Thank you,

Carl