Indigo server hangs

[jay (support)]

If I hazard a guess, it's unable to fully kill the hung IS process.

Seems unlikely given that IS continues to process - it's only network communication that seems to be a problem with indigo when in this state.

What is the exact OS version you're running?

Zombie processes, particularly ones that have somehow screwed up the network stack, will cause reboots to fail (since the process refuses to die). This would also explain all the symptoms everyone has been seeing. Something is taking down the OS network stack. Now we just need to figure out what, then why.

[jay (support)]

All: I never got consistent responses from everyone. If you haven't already, I need to know the exact OS version running on your Indigo Server Mac. I.E: 10.11.6 (15G1217) (you can get that from the System Information App when you select the Software header in the outline view to the left).

Next, if anyone notices their Mac in this state, try opening a terminal window (maybe just leave it running in the background so you don't have to launch it) and trying:

Code: Select all

ps aux | grep 'Z'

Then post the results here. This is what it looks like for me:

Code: Select all

USER               PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
jay              83262   0.1  0.1  2609712  21188   ??  S    Tue05PM   6:49.24 /Library/Application Support/Perceptive Automation/Indigo 7/IndigoPluginHost.app/Contents/MacOS/IndigoPluginHost -p1176 -fZ-Wave.indigoPlugin
jay              25380   0.0  0.0  2434840    796 s002  S+    9:50AM   0:00.00 grep Z


What we're looking for is any process which has a 'Z' in the STAT column - this indicates a zombie process.

I'm running Sierra 10.12.2.

IS has stayed up since this morning now, with HB disabled.

All: I never got consistent responses from everyone. If you haven't already, I need to know the exact OS version running on your Indigo Server Mac. I.E: 10.11.6 (15G1217) (you can get that from the System Information App when you select the Software header in the outline view to the left).

Next, if anyone notices their Mac in this state, try opening a terminal window (maybe just leave it running in the background so you don't have to launch it) and trying:

Code: Select all

ps aux | grep 'Z'

Then post the results here. This is what it looks like for me:

Code: Select all

USER               PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
jay              83262   0.1  0.1  2609712  21188   ??  S    Tue05PM   6:49.24 /Library/Application Support/Perceptive Automation/Indigo 7/IndigoPluginHost.app/Contents/MacOS/IndigoPluginHost -p1176 -fZ-Wave.indigoPlugin
jay              25380   0.0  0.0  2434840    796 s002  S+    9:50AM   0:00.00 grep Z


What we're looking for is any process which has a 'Z' in the STAT column - this indicates a zombie process.

Jay,
Here is my OS version: macOS 10.12.2 (16C67)
I managed to be sitting in front of the server when it hung. Lasted since 8:30 AM with everything running. Here is the grep z information you requested.

Grep Z 2017-02-02 at 4.13.52 PM.jpg (66.95 KiB) Viewed 3155 times

What I need to see to be able to help is the homebridge log, not the plugin log. There should be a "Show Homebridge Log" option in HBB.

Webdeck,
I cannot supply the HBB Plugin "Show Homebridge Log" as you requested as the event log is frozen when the server is in the hung condition. I tried from a remote Indigo 7 client also. Is a file available of the log somewhere that can be accessed?
Thanks for everyone help with this. I am going to shut all plugins and apps down except for HBB and see how that goes.
Jody

Webdeck,
I cannot supply the HBB Plugin "Show Homebridge Log" as you requested as the event log is frozen when the server is in the hung condition. I tried from a remote Indigo 7 client also. Is a file available of the log somewhere that can be accessed?
Thanks for everyone help with this. I am going to shut all plugins and apps down except for HBB and see how that goes.
Jody

Look in this directory:

Code: Select all

/Library/Application Support/Perceptive Automation/Indigo 7/Plugins/EPS Homebridge.indigoPlugin/Contents/Server Plugin/bin/hb/homebridge/

There should be a subdirectory in there named with the device ID of the HBB homebridge device. Inside that subdirectory will be a homebridge.log file.

Friday Morning Update:
Restating most of this past weeks observations...
My system hung Thursday afternoon 2/2/17 @4:39 PM
I restarted the system from a power on
I immediately closed all apps except for Terminal, Activity Monitor and Indigo Client
I disabled all user selectable plugins except for EPS Homebridge (HBB)
The only IndigoPluginHost running according to Activity Monitor are:
ZWave, Virtual Devices, Insteon Commands, EPS Homebridge, Action Collection and Script Executor
System hung at this morning 2/3/17 @4:06 AM (last Event Log update on Indigo Client running on Indigo Server Mac Mini)
No access via Indigo Touch
No zombies shown on "ps aux"
"node" showing constant stuck on "top" terminal command
Can connect via remote Indigo Client on my iMac - Event log still updating and can control Insteon devices
Zwave devices not working

I have attached the homebidge log. I am not sure why the last entry is 10:30PM

Thanks,
Jody

I'm at 24 hours up with HBB disabled.

I have attached the homebidge log. I am not sure why the last entry is 10:30PM

Thanks. I don't know why homebridge stopped doing anything at 10:03pm - there are no errors listed there, it just stops. The only errors I see are at startup because you have two devices with a "/" in their names - I/O Link #1 and I/O Link #2, which due to a RESTful API bug, are not escaped properly in the JSON responses. But that shouldn't harm anything.

How are you using homebridge? Are you interacting with it through the iOS Home app or via Siri or both? Are you using Homekit a lot for updating devices - I see a lot of updates, so I'm assuming you are using it pretty extensively. Do you remember doing anything around 10:03pm?

I want you to try something - I see on the homebridge project issues list, that Apple changed how Homekit locks work, and it is causing problems for some people. I see that you have a Basement Door Lock. Can you try excluding that device from homebridge to see if it fixes the issue?

-Mike

Mike,

Thanks for your reply.

I can't remember doing anything specifically at 10:03. I use both Siri and the Home App. I have been using it quite a bit just to keep making sure it isn't hung.

I will restart the server and make the change to the lock. I am not sure if this makes a difference but that "lock" isn't a real lock. It is a Virtual On/Off device. Just a contact switch on the dead bolt monitored by a Phidgets board.

When I bring the server back up, should I leave it in the barebones condition with just HBB plugin running?

Jody

Mike,

Thanks for your reply.

I can't remember doing anything specifically at 10:03. I use both Siri and the Home App. I have been using it quite a bit just to keep making sure it isn't hung.

I will restart the server and make the change to the lock. I am not sure if this makes a difference but that "lock" isn't a real lock. It is a Virtual On/Off device. Just a contact switch on the dead bolt monitored by a Phidgets board.

When I bring the server back up, should I leave it in the barebones condition with just HBB plugin running?

Jody

I would leave it as just HBB plugin running, to eliminate all other variables. Make sure homebridge gets restarted after excluding the lock. The issue we're testing with this is how Homebridge and Homekit are communicating with each other about the lock, so it doesn't matter what kind of device it actually is on the Indigo side. This is the issue that is referring to locks specifically: https://github.com/nfarina/homebridge/issues/849#issuecomment-276964669

For those of you seeing a node process that's either using excessive CPU or is in its zombie state, it would help to see the output from this command in a Terminal window:

Code: Select all

ps auxww | grep node

If we're lucky, this output will show what path/module the node utility is running. Unfortunately Activity Monitor cannot show this information without additional sleuthing through its Open Files and Ports tab for the matching process.

Note that if the node process is really a zombie that no useful data will be returned — since the process is just a stub waiting to give its return status back to its parent.

Just to add something to the conversation, I don't think my Indigo server has ever crashed or hung. Yesterday afternoon I loaded HBB for the first time ever configured it and everything worked well. I already had Home bridge running successfully, and just added HBB. I turned on my fan in the master bedroom last night from Indigo touch before going to bed so know that everything was working. When I woke up this morning nothing was working, the Indigo server was dead, (in a hung state that is). It would not restart. I removed the HBB plugin and restarted the box. Everything worked fine after that. The installation of HBB was the only change yesterday.

Terry


Sent from my iPhone using Tapatalk

[jay (support)]

Terry, what is the exact OS version you're running?

Sierra 10.12.3, on a Late 2012 Mini with 16GB, Jay. Indigo 7.0.2 at the time of the problems, upgraded to 7.0.3 afterwards though. Node 6.9.4 and the latest homebridge stuff, but I doubt it was that stuff because it's been running for several weeks.

Terry


Sent from my iPhone using Tapatalk

[jay (support)]

Everyone experiencing the issue has been running 10.12.2 or 10.12.3...

@webdeck & @colorado4wheeler, are either of you running Sierra? I'm not, and it's been running since last Wednesday without issue.