Indigo 7.5 Server mysteriously stops responding occasionally

Hi all,

I've noticed a few times that Indigo 7.5's server will simply stop responding to the client on my 8GB Mac mini M1 running Big Sur. It's not clear to me what's causing this, but it usually happens overnight, and when I come in the morning, the client is running but unable to connect to the server. A quick reboot fixes it, but it's happened three times so far.

Anyone else seeing this?

[matt (support)]

Haven't heard other reports of this (yet!). Do the Event Log files show any clues around the time it stops working?

Kinda' sorta' have a similar problem. I'll be out of the house for a bit... come back and the client computer has disconnected from the server and asking to reconnect.

I'll try to search the log next time it happens. Not really a big issue... its a one click fix... to reconnect...

Stay tuned.

Sooooo I don't have an answer, and not even sure where to look. Once again, when I got home from being away for the afternoon, the iMac running the Indigo client had disconnected from the server. Not sure what to look for in the log, but a quick glance didn't show anything unusual.

I actually searched for

Code: Select all

Indigo Cocoa client disconnected (192.168.5.121)

but didn't find one in the log from the time I left to the time I got home. Not sure what that means, but at least its some info. If/when it happens again, I'll see if I can find anything... but as I've said, this is hardly worth it. A single click of the mouse gets the client connected... no big deal.

It just happened again after working for days. Here's what I see:

Screen Shot 2021-01-21 at 5.49.11 PM.png (39.94 KiB) Viewed 2986 times

Is there any additional debugging I can supply before I reboot?

Just to be clear, is this happening on your computer running the server? or is it happening on a computer that you are just running the client on ? If the latter, you should be able to just click "Connect To Remote Server" and be good to go. I've seen this very very infrequently, but it has happened. I have never seen this on the computer that is actually running the Indigo Server.

The client and server are running on the same computer (the Mac mini M1).

Not sure if it is 7.5 or Big Sur.

BUT, HUGE problem with Server Not Responding.... crashes my plugin (EasyDAQ adaptation) all the time.

Here's an example, but randomly crashes when my CommThread calls the server:

2021-05-18 20:36:39.844 DEBUG Plugin.debugLog: <<-- entering deviceStartComm: p-Therm Spa (175318665 - haywardThermostat)
2021-05-18 20:36:39.844 DEBUG Plugin.debugLog: exiting deviceStartComm1 -->>
2021-05-18 20:36:39.845 DEBUG Plugin.debugLog: <<-- entering deviceStartComm: p-Water Feature (1143080645 - haywardOutputMirror)
2021-05-18 20:36:39.845 DEBUG Plugin.debugLog: added mirror of "*p-Hayward PL-PS-16" channel 5 to "p-Water Feature"
2021-05-18 20:36:39.846 DEBUG Plugin.debugLog: exiting deviceStartComm1 -->>
2021-05-18 21:18:22.944 ERROR Plugin.exceptionLog: Error in plugin execution:

Traceback (most recent call last):
File "/Library/Application Support/Perceptive Automation/Indigo 7.5/Plugins/Hayward.indigoPlugin/Contents/Server Plugin/hayward.py", line 923, in commThreadFunc
self._pool_actions(conn, dev, devProps)
File "/Library/Application Support/Perceptive Automation/Indigo 7.5/Plugins/Hayward.indigoPlugin/Contents/Server Plugin/hayward.py", line 729, in _pool_actions
_check_for_updates_from_thermostats()
File "/Library/Application Support/Perceptive Automation/Indigo 7.5/Plugins/Hayward.indigoPlugin/Contents/Server Plugin/hayward.py", line 684, in _check_for_updates_from_thermostats
for therm_dev in indigo.devices.iter("self"):
StandardError: ServerCommunicationError -- timeout waiting for response

2021-05-18 22:14:34.512 DEBUG Plugin.debugLog: <<-- entering deviceStopComm: *p-Hayward PL-PS-16 (1006890268 - haywardPool16)
2021-05-18 22:14:34.514 DEBUG Plugin.debugLog: exiting deviceStopComm -->>
2021-05-18 22:14:34.514 DEBUG Plugin.debugLog: <<-- entering deviceStopComm: p-Cleaner (1532647999 - haywardOutputMirror)
2021-05-18 22:14:34.515 DEBUG Plugin.debugLog: removed mirror of "*p-Hayward PL-PS-16" channel 3 to "p-Cleaner"
2021-05-18 22:14:34.515 DEBUG Plugin.debugLog: exiting deviceStopComm -->>
2021-05-18 22:14:34.515 DEBUG Plugin.debugLog: <<-- entering deviceStopComm: p-Filter (645712460 - haywardOutputMirror)
2021-05-18 22:14:34.516 DEBUG Plugin.debugLog: removed mirror of "*p-Hayward PL-PS-16" channel 17 to "p-Filter"
2021-05-18 22:14:34.516 DEBUG Plugin.debugLog: exiting deviceStopComm -->>
2021-05-18 22:14:34.516 DEBUG Plugin.debugLog: <<-- entering deviceStopComm: p-Firepit (519735858 - haywardOutputMirror)
2021-05-18 22:14:34.517 DEBUG Plu

[matt (support)]

Does the server eventually start working, or does it remain hung once it gets into this state? And does it ever actually crash, or just hang?

You might have a script or plugin that is causing the Indigo Server to spin off into the weeds. If it gets hung up launch the Activity Monitor app and see what processes are using a lot of CPU. If it is an IndigoPluginHost process, then you can click on the Files tab and should be able to see the plugin name.

So I have an adaptation of the EasyDAQ plugin rewritten to handle my pool (that is Hayward). I'm thinking something is off in there, as that is the plugin (its actually the Comm thread) that causes the timeout when I make a server call.

Couple of things/questions. I'm beginning to think something is pounding the server from that thread. Traffic is about a hundred packets a second between the plugin and the server. I know there should be quite a bit of traffic, but is that too much? And is there any way to see that traffic without putting a debug wrapper around every server call?

Appreciate the help here, because this is driving me nuts!

ns. I'm beginning to think something is pounding the server from that thread. Traffic is about a hundred packets a second between the plugin and the server. I know there should be quite a bit of traffic, but is that too much? And is there any way to see that traffic without putting a debug wrapper around every server call?

That's a crazy amount of traffic between the plugin and the server. What are you doing that needs to push data that often?

[jay (support)]

Yeah, that seems quite excessive...

Mine crashes as well on M1. I am not using EasyDAQ though.
Seems to panic on the server process.

I have disabled YouTube-dl and since then no crash happened.

Hope the new version fixes that anyway.

Ready for the release?

[matt (support)]

We will be releasing a new version that is M1 native soon, but that won't necessarily fix your crash as we don't have any reports of problems running Indigo 7.5 on M1 through Rosetta. I don't have any idea what YouTube-dl is.

It is a python library that is called as an external script.

It might have to do with Rosetta anyway.
Well, now you have two reports.