10-05-2017 02:16 AM
Some times when I launch the flow manager , I'm getting the below error as
"Error communicating with the daemon : Invalid response - Error communicating with daemon".
When I tried the check the log, I could not able to figure out any errors in below log files
The above log files are found under LSF log folder. When I usually gets this error, I will restart the whole application and then the issue gets resolved.
Could not able to figure out the exact reason for this. This is affecting our overnight batches and that causes a delay. When i restart the application during business hours, its affecting the Business (the metadata server goes down and resumes back.)
We are using Platform Process Manager 8.1 and SAS 9.1.3 version. Could you please help me to find out the fix for this issue.
Thanks in advance.
10-05-2017 02:36 AM
Yes, I've had this before (....still have it).
But first, let's see if it's the same problem. I'm assuming UNIX here.
ps -ef | grep -v grep | grep jfd
...and see if there are more than one jfd process. There should normally just be one. However, it is normal for to have a second jfd process (spawned by the first jfd) for a brief period of time. If the second jfd persists, it might be the same issue as us. Like you say, when you kill and restart, all is ok.
If the second jfd persists, check your logging level in lsf.conf (..I think). For some reason, our log level was set to DEBUG causing excessive logging during busy periods. I have reduced the logging level to normal and so far so good. IBM has provided us with a patch and a new jfd executable that will log additional traces for them to investigate, but since we're in freeze, I haven't been able to apply it.
If you have the same issue, I have a Track # that I can share which will save you and Tech Support many hours of troubleshooting.
10-05-2017 02:59 AM
I tried running the ps -ef | grep -v grep | grep jfd in my UNIX server where sas has been installed.
I could able to see only one jfd process under root user.
10-05-2017 02:46 AM
Just one point to note. Flow Manager will be an issue with Process Manager (server side), not LSF. You mentioned you found the logs under ../lsf/log. Look under ../pm/log for the jfd* log
10-05-2017 03:25 AM - edited 10-05-2017 03:35 AM
Thanks for pointing out to jfd log.
I have got the issue on Oct 3,2017 at around 04:30 and have the log as
JFFLowmanger:: ProcessFlowEvent : Unable to send mail for flow
That's the last log entry in the jfd log file at 04:30 and then I have restarted at 09:22 and we have entry as
JFuser :: initLsfUserDomain LSF_user_Domain =
JS_ARM_Enable = False
JFLockFIile ::: getlock : Got the lock file
This process manager is processed with SAS License
JFArchive :: _readByte : reading beyond end of file.
I don't see any error entries in the log file at around 04:30.
10-05-2017 04:45 AM
My apologies, I should have mentioned. You run the command WHEN you are having the issue and BEFORE you restart Process Manager.
Also, you mentioned SAS 9.1.3. Why such an old version ?
10-05-2017 04:58 AM
I also implemented a monitoring script that checks for multiple jfd processes every minute. This way you can get onto it before getting complaints. Just a word of caution. It is normal for the first jfd to launch another jfd briefly...as per IBM. But it's quick. If the second one persists, then you have the issue i.e. you might some false positives sometimes.
10-05-2017 05:21 AM - edited 10-05-2017 05:22 AM
Let me try out the script when the issue persists again..Hopefully, it should not
We are migrating to 9.4 and will be completed soon.
So, that means there is no fix for this issue. Every time , we have to restart the application, if the issue occurs.
10-05-2017 05:38 AM
We're still working on the fix. The Tech Support Track # is 7612222597. Maybe log a Track and make reference to mine and say the symptoms are similar.
What is the value of LSF_LOG_MASK in ../lsf/conf/lsf.conf ? Ours was set to to LOG_DEBUG which I believed caused the problem under heavy load. Generally logging should only be increased to DEBUG for a short period while troubleshooting. Definitely not under normal operations. Still to be confirmed, but I believe this was a contributor.
10-05-2017 06:00 AM
Ok, then I suggest you log a call and maybe they can give you the jfd module that allows for additional tracing. IBM will analyse this trace to see what's going on.
11-03-2017 02:42 AM
Unfortunately I had to handover the issue to another resource and I'm not sure if they were able to resolve it.
My only suggestion is to troubleshoot with Tech Support....
11-03-2017 03:40 PM
SAS has notice of this problem on the IBM product since 2015. http://support.sas.com/kb/57/006.html
Please check this note. Perhaps your Server minor version is newer than your client version. This might be the case is there was an update on the server side, but never happened on the client side. Anyway, 2 things worth from it:
1. try this fix.
2. Install patches IBM on LSF on server and client side. Always with the help of SAS Technical Support. There are many issues on LSF solved by installing a few patches.