topic Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"? in Administration and Deployment

Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

rgreen33 — Tue, 14 Mar 2017 14:07:39 GMT

We are attempting to initialize and enable the Service Architecture in our SAS environment (Distributed with Hadoop - Linux). I am following this documentation (http://support.sas.com/documentation/cdl/en/evug/69029/HTML/default/viewer.htm#n18wwqg60hkjqon12qo39m19e1pf.htm ). We have successfully initialized the SAS Environment Manager Extended Monitoring. We have successfully Enabled and Initialized the APM ETL. Now, we are on the last step for Enabling ACM ETL. When we attempt to run the command "./emi_init.sh --enable ACM", it is hanging at "Starting metricData".

Here's the output from the command:

[sas@sasdr bin]$ ./emi_init.sh --enable ACM

Service Architecture Initialization Utility - Tue Mar 14 08:56:05 EDT 2017

Validating environment...

Validating external connections...

ACM Enabled

Running ACM ETL...

Service Architecture ACM ETL Process - Tue Mar 14 08:56:19 EDT 2017

Starting dbMirrorData...

Starting inventoryUpdate...

Service Architecture Group Creation - Tue Mar 14 08:56:31 EDT 2017

Creating/updating groups...

Successfully processed 11 groups in 25.346 sec

Starting metricData

And it just hangs here...for hours.

The end of the log file looks like this:

***Log file getMetricData_20170314_085619.sas.log***

NOTE: Appending WORK.HQ_METRIC_DATA to EVDMRAW.HQ_METRIC_DATA.

NOTE: There were 0 observations read from the data set WORK.HQ_METRIC_DATA.

NOTE: 0 observations added.

NOTE: The data set EVDMRAW.HQ_METRIC_DATA has 1 observations and 3 variables.

NOTE: PROCEDURE APPEND used (Total process time):

real time 0.00 seconds

user cpu time 0.00 seconds

system cpu time 0.00 seconds

memory 565.65k

OS Memory 18088.00k

Timestamp 03/14/2017 08:57:06 AM

Step Count 18 Switch Count 0

Page Faults 0

Page Reclaims 55

Page Swaps 0

Voluntary Context Switches 0

Involuntary Context Switches 0

Block Input Operations 0

Block Output Operations 144

^L8 The SAS System 08:57 Tuesday, March 14, 2017

NOTE: DATA statement used (Total process time):

real time 0.00 seconds

user cpu time 0.00 seconds

system cpu time 0.00 seconds

memory 806.59k

OS Memory 18088.00k

Timestamp 03/14/2017 08:57:06 AM

Step Count 19 Switch Count 0

Page Faults 0

Page Reclaims 221

Page Reclaims 5

Page Swaps 0

Voluntary Context Switches 0

Involuntary Context Switches 0

Block Input Operations 0

Block Output Operations 8

NOTE: DATA statement used (Total process time):

real time 0.00 seconds

user cpu time 0.00 seconds

system cpu time 0.00 seconds

memory 805.68k

OS Memory 17832.00k

Timestamp 03/14/2017 08:57:06 AM

Step Count 47 Switch Count 0

Page Faults 0

Page Reclaims 217

Page Swaps 0

^L18 The SAS System 08:57 Tuesday, March 14, 2017

Voluntary Context Switches 0

Involuntary Context Switches 0

Block Input Operations 0

Block Output Operations 0

NOTE: DATA statement used (Total process time):

real time 0.01 seconds

user cpu time 0.01 seconds

system cpu time 0.01 seconds

memory 816.28k

OS Memory 17832.00k

Timestamp 03/14/2017 08:57:06 AM

Step Count 48 Switch Count 51

Page Faults 0

Page Reclaims 220

Page Swaps 0

Voluntary Context Switches 330

Involuntary Context Switches 1

Block Input Operations 0

Block Output Operations 0

What am I missing?

Thanks,

Ricky

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

JuanS_OCS — Tue, 14 Mar 2017 14:44:08 GMT

Hello @rgreen33,

if you have big amount (or size) of logs from your SAS services, and from your SAS Environment Agent (the data folder), it could be that it is processing all of this data.

Remember: The SAS logs will not write a running data step, until the data step is complete. So, the last data step you could see, is the last completed data step.... but if you go to the related SAS code that is being executed, most probably you will see addtiional data steps or procedures, and the first one will the the one currently running. I can easily imagine that it is a process that is loading data from the logs or your agent metrics into the WIP database (your EVDM library).

You can also check the running sas.exe processes and the activity on your server resources.

Just for fun and giggles, it could be interesting to enable emi_init after a clean/archive of your logs and agent data?

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

rgreen33 — Tue, 14 Mar 2017 14:52:31 GMT

@JuanS_OCS,

Thank you for your reply. We initially kicked the process off around 3:00 PM yesterday. This morning, it still appeared to be hanging. Watching the log file and the files under Datamart folders, we are not seeing any activity. It goes to a certain point and then appears to hang. We killed the process this morning and kicked it off again...same thing happened. 😞 Watching the service in Environment Manager, it goes from red to green to yellow and then back to red. Do you know if there is a way to see if it is truly still active? Anything that I can watch to see the activity?

Thank you,

Ricky

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

JuanS_OCS — Tue, 14 Mar 2017 14:59:46 GMT

Hello @rgreen33,

I have seen similar behaviour on other processes, and on this process trying to access network resources without any success.

http://support.sas.com/documentation/cdl/en/evug/69029/HTML/default/viewer.htm#p0jgzt0ahiohlqn1c9auz89glv0e.htm

This link explains the pre-requisites, and some of them speak about the SSL and network shares. Did you have the opportunity to check all of them?

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

rgreen33 — Tue, 14 Mar 2017 15:40:31 GMT

@JuanS_OCS,

Thanks again for your help. After re-reading the prerequisites, I may have found the issue. Per the prerequisites...

Ensure that your SAS deployment contains a functioning Stored Process server. The server must be defined on the SAS Environment Manager Enablement Kit Server, and it cannot be load balanced across multiple machines.

Looking more closely at my environment, I found that I may have Stored Process Server load balanced across multiple machines. Here's what my SASApp - Stored Process Server looks like in SAS Management Console:

Could this be my issue? If so, does that mean that I will NOT be able to run ACM? What about APM? It looks as if everything is working fine with APM. Should I disable it since the SASApp - Stored Process Server is load balanced?

Thanks again for your help,

Ricky

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

JuanS_OCS — Tue, 14 Mar 2017 15:54:25 GMT

Hi there @rgreen33,

no, your Stored Process server is completely normal. it is load balanced (by default, as it should be) and in a single server (it is possible to configure the Stored Process to run on multiple servers). So this one should not be your problem.

From https://support.sas.com/documentation/cdl/en/evug/69029/PDF/default/evug.pdf

Working with ACM ETL
The ACM ETL process extracts and stores metric data that is collected by the SAS
Environment Manager agents into the data mart. Storing the data in the data mart
enables you to access the data for a longer period of time than if it was stored only in
the SAS Environment Manager web application.
ACM data is processed and loaded into the data mart in these steps:
1 Metric data is collected from the SAS Environment Manager agents and sent to the
SAS Environment Manager database.
2 At specified intervals, the ACM ETL process runs. The process copies data from the
database, standardizes the data, and loads the data into the data mart.

So, I still think it is a question related to the agents. A couple of checks I would propose you:

If you go to the SAS Environment Manager, to Resources, are all of your Platforms with green check marks?
What is the total size in GB of every /Web/SASEnvironmentManager/agent-XXXXX/data folder of every node on your deployment?
What is the size of your ACM tables? http://support.sas.com/documentation/cdl/en/evug/69029/HTML/default/viewer.htm#n13wvzy0q896tsn19cqmssxv1pza.htm

These tables are located in the directory [levelroot]/Web/SASEnvironmentManager/emi-framework/Datamart/acm.
ACM.AVAILABILITY Table
ACM.FILEMOUNTS Table
ACM.GROUPINVENTORY Table
ACM.EVENTS Table
ACM.HOSTPLATFORMS Table
ACM.HTTPCHECKS Table
ACM.IOMSERVERS Table
ACM.MEASUREINVENTORY Table
ACM.METADATASVRS Table
ACM.NETWORKINTERFACE Table
ACM.RESOURCEINVENTORY Table
ACM.SASAPPSERVERS Table
ACM.SASLOGICALSERVERS Table
ACM.TCSSERVERMGRS Table
ACM.WEBAPPSERVER Table
ACM.WIPDATADB Table

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

rgreen33 — Tue, 14 Mar 2017 18:32:50 GMT

@JuanS_OCS,

So, I checked the Environment Manager Resouces...and all are green (2 of them).

I checked on the size of the files in our /Web/SASEnvironmentManager/agent-XXXXX/data folder. Here are the results:

[sas@sasdr data]$ du -hs /data/sasconfig/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/data

37M /data/sasconfig/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/data

I also checked the file of our Datamart acm files. We only have 4 files in that folder:

Does this help?

Thanks,

Ricky

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

rgreen33 — Tue, 14 Mar 2017 18:49:04 GMT

@JuanS_OCS,

One thing to add...this is a fresh install. The install of our servers "completed" and we are just now getting users setup and running. We are working out some of these "kinks" before we let too many people get on the systems. One thing to note...not sure if it is related to this or not...but, we do have several ActiveMQ 5.7 Topic services that are currently showing up in Environment manager as "unavailable". We are just trying to tackle one issue at a time.

Thanks,

Ricky

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

JuanS_OCS — Thu, 16 Mar 2017 09:14:07 GMT

Hello @rgreen33,

I am practically sure that it is related to the agents. In case you have inactive ActiveMQ services, you probably might want to understand what is going on those.

Could you please send a screenshot of your SAS Environment Manager, the Resources page, and the full list of Servers tab?

What errors can you find on the logs of the ActiveMQ?

Also, it might be an idea to start a track with your SAS Technical Support in pararell/

Thank you in advance.

Juan

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

chardin — Thu, 16 Mar 2017 15:38:03 GMT

Juan,

I'm adding some attachments on Ricky's behalf, while he is away. I work with him in supporting the same SAS environment.

One of the screenshots provides a list view on all the EM Resources - Servers. The other shows a list view of all the EM Resources - Unavailable Services as well. The only error I found in the ActiveMQ log file (/data/sasconfig/Lev1/Web/activemq/data/activemq.log) was dated Nov. 28, 2016. The same log contains several warning messages which are more current and shows numerous failed connections. I’ve included Word documents for each of those.

Please note that a SAS Suport ticket/track has been opened on this issue as well. I hope this information helps.

Thank you for the additional assistance.

Carl

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

chardin — Thu, 16 Mar 2017 15:44:07 GMT

Attachment 2 showing the ActiveMQ error...

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

chardin — Thu, 16 Mar 2017 15:45:35 GMT

Attachment 3 showing SAS Environment Manager - Resources - All Servers...

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

chardin — Thu, 16 Mar 2017 15:46:49 GMT

Attachment 4 showing SAS Environment Manager - Resources - Unavailable Services...

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

JuanS_OCS — Thu, 16 Mar 2017 16:07:21 GMT

Hello @chardin,

thank you for taking over @rgreen33 's communications!

What I can see here, is that your ActiveMQ connections (your JMS Broker, the metrics collector, I guess) is starting when there is a same process already started, blocking the port number.... or your firewall is giving the problems. Maybe the services were not stoped properly and there are some roge processes that are blocking your ports. In this case, netstat and ps are our commands, then kill the processes.

Once you do this, and you ensure no SAS processes are started, just try to start al the services on the nodes.

Could you please attach the 4th attachment? Something was wrong when you tried to post it.

PS. Something that might help:

For firewall and network issues:

http://support.sas.com/kb/43/946.html

http://support.sas.com/kb/52/670.html

Solve ActiveMQ errors:

http://support.sas.com/kb/54/066.html

http://support.sas.com/kb/52/315.html

Re: Why is "./emi_init.sh --enable ACM" hanging on "Starting metricData"?

chardin — Thu, 16 Mar 2017 17:32:16 GMT

Juan,

Thanks for the URL's. I'll review them next. Here is the missing attachment showing the unavailable services.

Thank you,

Carl