Indigo Server Subscribe/Broadcast Dropping Messages

I have been utilizing the Indigo broadcast/subscribe feature in my plugins for a few years now, but I have always had these phantom issues that I could never track down.

Specifically, I developed ShellyMQTT that interfaces with Shelly devices over MQTT. FlyingDiver's MQTTConnector plugin manages the MQTT portion of the solution and passes MQTT messages to ShellyMQTT through the broadcast/subscribe "pipe". I have always run into 1-in-a-million bugs where I see MQTTConnector processed a message and ShellyMQTT never received it. Of course, the next message goes through and I have no idea what just happened.

Fast forward to recently - I wrote a script that processes Webhooks sent from Plex. This script then packages up the data onto an MQTT topic and publishes it to my broker. Now I can have fun making Indigo control lighting when I play/pause/resume/stop content playing on my TV. But this is unreliable where 1-in-4 messages fail to make it through. I clearly see all messages being received by MQTTConnector and it is configured correctly to publish this message for another plugin to process it. But the other plugin fails to receive this message.

Question: Is there some sort of internal buffer in Indigo that could potentially be dropping messages? My mature MQTTConnector -> ShellyMQTT flow routinely sees bursts (60+ within milliseconds) and occasionally some will get dropped. These are very small messages averaging maybe 100 bytes. The Plex messages routinely being dropped are about 32Kb in size (JSON).

I would first suspect that your plugin isn't processing the message fast enough and two more come in before you can grab the next one, so one is lost. I would implement a queue system so that you can return from the broadcast call as fast as possible.

I would first suspect that your plugin isn't processing the message fast enough and two more come in before you can grab the next one, so one is lost. I would implement a queue system so that you can return from the broadcast call as fast as possible.

So in the latest case, it can fail to process a message with no other activity. Simply publishing one of these large Plex will sometimes get lost. And this is using MQTTConnector and the Shims plugin. I can clearly see the message processed by MQTTConnector in the log and then Shims does not pick it up. But it would pick up the next one and a handful after that.

I was just able to finally find this occurring across multiple plugins where messages seem to be dropped when sending to ShellyMQTT or to Shims. The subscribe/broadcast pattern is the common link here.

[matt (support)]

Are there any errors logged in the Event Log (from the server or plugin host) about the command or packet being corrupt?

I'm not sure what is going wrong here – internally, the mechanism used here is the same one Indigo uses for all messages coming from the Indigo Server (for device state updates, action callbacks, etc.), so it is pretty thoroughly tested.

Note all of the callbacks into the plugin happen from the same thread. So if you had a plugin action callback (or some other callback) that was hung up processing for some reason, then that would delay when the subscribed broadcast callback would execute. But it should always still execute (presuming the other callback eventually completes). The internal messaging queue is unlimited, so the plugin will crash before it drops a message. The only time a message would be missed would be if the plugin wasn't running because it was restarted or crashed (both of which would show in the Event Log).

Lastly, on the off chance the failure is related to the contents of what is being broadcast (ex: it is really large), you might try settings up a broadcast/subscribe for a simple "ping" style command and see if those always are getting through or not.

So I figured it out (face_palm). All messages were/are being correctly passed by Indigo. Not surprising given it is Indigo...

MQTTConnector was receiving messages from the broker and correctly adding the messages to the relevant internal queues based on each trigger pattern. ShellyMQTT had been the sole consumer of messages broadcast by MQTTConnector until this test.

ShellyMQTT had a half-completed feature that would explore all Indigo triggers and extract all the message types that the user had defined. This was supposed to make it easier for the user to select the message type when creating a device since they wouldn't need to go look it up. The problem is that, for some reason, I was saving all the message types to the same variable that was being used to determine whether or not to process incoming message types.

For example, I have 2 triggers:

1. Trigger A matches on "shellies/#" and queues messages with a message type of "shellies"
2. Trigger B matches on "plex/#" and queues messages with a message type of "plex"

ShellyMQTT should scan all its own devices and determine which message types are used by its devices. This would just be the message type of "shellies". However, the plugin was also examining triggers and it incorrectly added the message type of "plex" to the wrong internal message_type variable. When a plex message was queued, MQTTConnector queued the message in the "plex" queue and broadcast to everyone that there is a message in the plex queue. ShellyMQTT was being greedy and consuming these messages if it processed the notification before the MQTT Shims plugin.

This made it seem as if the message was being dropped between MQTTConnector and MQTT Shims since ShellyMQTT was stealing them and sending them to the abyss.