Errors on timeouts

I seem to have started getting occasional timeout messages to my Shelly plus 1 and Shelly 1 devices. Unfortunately, the plugins are not handling the message properly and I get syntax errors. True in both plugin versions (see below):

Code: Select all

Apr 15, 2023 at 8:44:43 AM
   ShellyNGMQTT Error              Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 78, in runConcurrentThread
Exception: ServerCommunicationError -- timeout waiting for response

   ShellyNGMQTT Error              plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
   ShellyMQTT Error                Error in plugin execution runConcurrentThread:

  File "plugin.py", line 330, in runConcurrentThread
type: ServerCommunicationError -- timeout waiting for response

   ShellyMQTT Error                plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)



The situation corrects itself after generating the log message, but I have been getting one message every other hour or so for the last day. And its always ShellyNGMQTT AND ShellyMQTT simultaneously.

That's not a syntax error. That's a server communication timeout error. The question is, Which server? The Indigo server? The MQTT server?

Is there any correlation with your Indigo server going to sleep? These errors are not from within the plugins, but the inter-process communication between the plugins and the MQTTConnector plugin. You likely don't see these errors with other plugins because there don't seem to be a lot of other plugins taking advantage of the inter-plugin communication.

Thanks for the responses. I am not sure how to track this down - either identifying the problem / cause nor the solution. Here is the best color I can share:

1) The sleep issue seems to have gone away since I did an SMC reset on Wednesday 4/12 in the morning. Having said that, I dont think enough time has passed for me to consider the issued solved - I would say two weeks without sleep means I made progress. Right now we are only at 3 days.

2) The MQTT server communication timeout errors started on 4/12 in the evening at about 11 PM - I did not notice until yesterday but I checked the logs to confirm. Timing goes back to when I did a complete reset of my router (ASUS RT-AX92U in Mesh mode with two wired AI nodes). I reset the router because it had been crashing every few days. It's quite possible that the router is failing. It's been behaving weirdly lately, like very slow page refreshes on its web administration server (192.168.1.1) = that issues seems to come and go. But the timing of the server communication timeout errors directly coincided with the router refresh.

3) I will also note that I started having issues about a month ago when my Internet was upgraded to fiber. The cable company's router blasts wifi which causes some interference with my home network. I see lots of timeouts periodically in my TP-link modules (using TP-Link plugin and Kasa light switches and dimmers). Could be related.

Having said that, is a router issue likely to cause server communication timeout errors with the MQTT plugin and nothing else?

Here is a bit more of my log that illustrates the timeout activity

Code: Select all

Apr 15, 2023 at 1:44:25 PM
   Received INSTEON                "Thermostat-Boys Room" temperature changed to 69.0

Apr 15, 2023 at 1:45:31 PM
   Received INSTEON                "Thermostat-Boys Room" temperature changed to 70.0
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Entry Chandelier on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Overhead #1 - S1 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Hanging - S2 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Living Room Hall - S2 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Foyer Sconce on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Overhead #1 - S2 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device kitchen Hanging - S1 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Overhead #2 - S1 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Wine Room
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Overhead #2 - S2 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Under Great Room
   TP-Link Devices                 2 timeouts - Normal polling resuming for device Basement Water Pump Room
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Exercise Room2
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Lower Stairs
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Exercise Room1
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Exercise Room Back
   TP-Link Devices                 2 timeouts - Normal polling resuming for device Basement Playroom Back pair
   TP-Link Devices                 2 timeouts - Normal polling resuming for device basement Lower Stairs Pair1
   TP-Link Devices                 2 timeouts - Normal polling resuming for device Basement Bathroom Overhead
   TP-Link Devices                 2 timeouts - Normal polling resuming for device Pool House Outside
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Playroom Front
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Entry Chandelier
   TP-Link Devices                 2 timeouts - Normal polling resuming for device basement Outside Light
   TP-Link Devices                 1 timeouts - Normal polling resuming for device kitchen Hanging
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces
   TP-Link Devices                 1 timeouts - Normal polling resuming for device guest bedroom back
   TP-Link Devices                 1 timeouts - Normal polling resuming for device TV Room Sconce
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Office Lights
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Overhead #2
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Overhead #1
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Playroom Back
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Foyer Sconce
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Foyer
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Master Bath Overhead
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Master Bath Mirror
   TP-Link Devices                 2 timeouts - Normal polling resuming for device TV Room Lights
   TP-Link Devices                 2 timeouts - Normal polling resuming for device guest bedroom front
   TP-Link Devices                 3 timeouts - Normal polling resuming for device Entry Chandelier on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Master Bath Mirror
   TP-Link Devices                 3 timeouts - Normal polling resuming for device Living Room Hall - S2 on/off
   TP-Link Devices                 3 timeouts - Normal polling resuming for device Kitchen Overhead #2 - S2 on/off
   TP-Link Devices                 3 timeouts - Normal polling resuming for device Kitchen Overhead #1 - S2 on/off
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Kitchen Overhead #2
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Entry Chandelier
   TP-Link Devices                 3 timeouts - Normal polling resuming for device kitchen Hanging - S1 on/off
   TP-Link Devices                 2 timeouts - Normal polling resuming for device Office Lights
   TP-Link Devices                 2 timeouts - Normal polling resuming for device Kitchen Overhead #1
   TP-Link Devices                 2 timeouts - Normal polling resuming for device kitchen Hanging
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Basement Water Pump Room
   TP-Link Devices                 4 timeouts - Normal polling resuming for device Kitchen Hanging - S2 on/off
   TP-Link Devices                 4 timeouts - Normal polling resuming for device Kitchen Overhead #2 - S1 on/off
   Received INSTEON                "Logan bedroom" on
   TP-Link Devices                 2 timeouts - Normal polling resuming for device Kitchen Foyer
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Entry Chandelier on/off
   TP-Link Devices                 6 timeouts - Normal polling resuming for device Foyer Sconce on/off
   TP-Link Devices                 6 timeouts - Normal polling resuming for device Step Sconces on/off
   TP-Link Devices                 7 timeouts - Normal polling resuming for device Kitchen Overhead #1 - S1 on/off
   TP-Link Devices                 3 timeouts - Normal polling resuming for device Step Sconces
   TP-Link Devices                 3 timeouts - Normal polling resuming for device Master Bath Overhead

Apr 15, 2023 at 1:46:38 PM
   Received INSTEON                "Thermostat-Boys Room" temperature changed to 69.0
   Error                           reflector connection test failed: unable to connect to server
   Warning                         reflector reconnection scheduled in 5 seconds
   Reflector                       restarting reflector connection to https://hamptons20mile.indigodomo.net

Apr 15, 2023 at 1:47:21 PM
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces

Apr 15, 2023 at 1:48:09 PM
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces

Apr 15, 2023 at 1:48:35 PM
   Received INSTEON                "Boys Bathroom Sconce" on
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces
   Received INSTEON                "Thermostat-Boys Room" temperature changed to 70.0

Apr 15, 2023 at 1:50:33 PM
   Indigo Cocoa client authenticated (192.168.1.62)

Apr 15, 2023 at 1:50:59 PM
   ShellyMQTT Error                Error in plugin execution runConcurrentThread:

  File "plugin.py", line 330, in runConcurrentThread
type: ServerCommunicationError -- timeout waiting for response

   ShellyNGMQTT Error              Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 78, in runConcurrentThread
Exception: ServerCommunicationError -- timeout waiting for response

   ShellyNGMQTT Error              plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
   ShellyMQTT Error                plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)

Apr 15, 2023 at 1:51:34 PM
   Indigo Cocoa client disconnected (192.168.1.62)

Apr 15, 2023 at 1:52:22 PM
   TP-Link Devices Error           (tpl_relay_poll.py:run:113): 460 consecutive polling errors for device TV Room Left BOTTOM Outlet: error TP-Link connection timeout. Polling internal now 47

Apr 15, 2023 at 1:57:02 PM
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces

Apr 15, 2023 at 1:57:59 PM
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces

Apr 15, 2023 at 1:58:48 PM
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces

Apr 15, 2023 at 1:59:13 PM
   Received INSTEON                "Thermostat-Master Bedroom" on (button 1)

Apr 15, 2023 at 1:59:47 PM
   Received INSTEON                "Thermostat-Master Bedroom" temperature changed to 72.0

Apr 15, 2023 at 2:01:23 PM
   Received INSTEON                "Boys Bathroom Fan" on

Apr 15, 2023 at 2:01:50 PM
   TP-Link Devices                 1 timeouts - Normal polling resuming for device Step Sconces

Apr 15, 2023 at 2:02:06 PM
   TP-Link Devices Error           (tpl_relay_poll.py:run:113): 470 consecutive polling errors for device TV Room Left BOTTOM Outlet: error TP-Link connection timeout. Polling internal now 48

Apr 15, 2023 at 2:03:51 PM
   Received INSTEON                "Thermostat-Boys Room" temperature changed to 69.0

Apr 15, 2023 at 2:04:06 PM
   Received INSTEON                "Thermostat-Master Bedroom" temperature changed to 71.0

Apr 15, 2023 at 2:08:45 PM
   Indigo Cocoa client authenticated (192.168.1.62)

Apr 15, 2023 at 2:09:03 PM
   ShellyMQTT Error                Error in plugin execution runConcurrentThread:

  File "plugin.py", line 330, in runConcurrentThread
type: ServerCommunicationError -- timeout waiting for response

   ShellyMQTT Error                plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
   ShellyNGMQTT Error              Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 78, in runConcurrentThread
Exception: ServerCommunicationError -- timeout waiting for response

   ShellyNGMQTT Error              plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)

Apr 15, 2023 at 2:11:24 PM
   Received INSTEON                "Boys Bathroom Fan" off
   TP-Link Devices Error           (tpl_relay_poll.py:run:113): 480 consecutive polling errors for device TV Room Left BOTTOM Outlet: error TP-Link connection timeout. Polling internal now 49

Apr 15, 2023 at 2:13:14 PM
   Received INSTEON                "Thermostat-Master Bedroom" off (button 1)

Apr 15, 2023 at 2:13:49 PM
   Received INSTEON                "Thermostat-Master Bedroom" temperature changed to 70.0


[jay (support)]

The errors you are seeing in the log often occur when there are networking issues on your Mac. Plugins communicate with the server over network sockets, so if the Mac's network stack gets into an odd state it causes strange errors. It also might be exacerbated by the TP Link plugin, which definitely seems to be having some issues talking to it's devices.

One thing to try would be to disable the TP Link plugin and see if those errors go away.

Stalled admin page

Screenshot 2023-04-15 at 3.39.53 PM.png (424.13 KiB) Viewed 1742 times

Jay - thanks for the suggestion. The TP-link plugin is just indicating it is have g timeout issues in communicating with the devices. And actually these issues are less frequent now than they have been for some time. The Optimum wifi (which I am not using but is needed to communicate with my TV STB's from the cable company) is interfering g with my 2.4 gHz home internet provided by the Asus network. I have had to move a series of devices off the Asus router and onto a new router connected by ethernet to the Asus (so basically I introduced a third wifi network because #2 was contaminating #1). At least the new WiFi is part of my network and not external to it like the Optimum signal. I can try to turn off TP-link plugin but I would be shocked if it had an effect.

What is weird about the MQTT related timeout is they occur together, about 20 times per day (per my log), but the Shelly devices that communicate MQTT and MQTT-NG are in different parts of the house, yet they timeout in sync.

The other thing worth noting is that the issues started when I reset the router. And since they I have had other issues with that router. See screenshot of where router stalled connecting to its administration page. It pretty ,much timed out there until I refreshed the page (after a couple of minutes eating). I then had to sign in again and it worked fine (see second screenshot for comparison).

Only other test I can think of, which I will have to do next week when I have more time, is to swap my base router with one of the Mesh nodes (they are the same units). If the router is crashing with a hardware problem, I would imagine this would correct it.

Just an update - I did two things yesterday. I reconnected a Tp-link outlet off the Asus router to the new one (eliminating the inference / weak signal issue) and I removed the USB drive that was attached to the router. I would not think either of these things would be corrective. But since 10:13 PM last night (13 hours now) there have been no timeouts affecting MQTT and very few on the TP Link devices (only one which is between the main router and one of the Mesh nodes). Obviously I will continue to monitor but the issue seems to have fixed itself.

I started getting these error messages again. It's odd that I only get error messages in Shelly MQTT scripts and nothing else. I had a laptop running Indigo client app that was left connected to the server when the laptop went to sleep. Laptop was sitting in sleep mode untouched, but periodically (while still sleeping) it seemed to connect ("Indigo Cocoa client authenticated (192.168.1.62)") and then generate the ShellyMQTT and ShellyNGMQTT timeout errors, then disconnect. Happened about 7 times over a twelve hour period until I grabbed the laptop, woke it up, quit the Indigo client, and put it back to sleep. Now these timeout messages have stopped. Very odd to me.

[edit] Just realized that this same issue with sleeping laptop went on the last time I had these message/timeouts. It looks like Indigo server periodically does some interaction with the sleeping laptop where I forgot to close Indigo client.
"

Code: Select all

Jun 4, 2023 at 10:43:09 AM
   Received INSTEON                "Thermostat-Boys Room" on (button 2)

Jun 4, 2023 at 10:46:38 AM
   Indigo Cocoa client authenticated (192.168.1.62)

Jun 4, 2023 at 10:46:54 AM
   ShellyMQTT Error                Error in plugin execution runConcurrentThread:

  File "plugin.py", line 330, in runConcurrentThread
type: ServerCommunicationError -- timeout waiting for response

   ShellyMQTT Error                plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
   ShellyNGMQTT Error              Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 78, in runConcurrentThread
Exception: ServerCommunicationError -- timeout waiting for response

   ShellyNGMQTT Error              plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)

Jun 4, 2023 at 10:47:40 AM
   Indigo Cocoa client disconnected (192.168.1.62)

Jun 4, 2023 at 10:50:49 AM
   Received INSTEON                "Thermostat-Boys Room" off (button 2)


Just in case anyone else sees these, this error correlates perfectly with when I connect my laptop to Indigo and then put laptop to sleep without disconnecting from Indigo first.