Message Size Limitation for MQTT Connector?

Hi, I'm relatively new to MQTT and am developing a new indigo pluin using the ringmqtt project. All the messages are processing nicely except for one type, it's playload is a binary image. I can see the message in MQTT Explorer and I've been able to capture and store the image using a different client connector external to indigo, so the MQTT Broker side is working correctly. The image is 43K but could vary from there slightly. And the adjacent attributes topic is coming through properly. I have even set up a trigger matching that specific topic and it never fires.

Any ideas?

I suspect you're running into some Indigo limit on the size of the content of a custom state. The connector plugin stores the incoming message topic and payload in a device state, and that size payload is probably just too large. I know I've never tried to use anything with a payload anywhere near that size. Honestly, that kind of payload seems contrary to the intent of the MQTT protocol.

I'd need some input from @Matt or @Jay to determine what's a reasonable upper limit for states. Also if there's a limit on the size of the payload for the broadcastToSubscribers call, which is how the MQTT messages get dispatched to other plugins. If it's just the state that's causing the issue, I can put in code to prevent large payloads from being saved to the state. But if broadcastToSubscribers is also limited, then I don't think using the MQTT Connector plugin is feasible. In which case you should use the paho library directly in your plugin and skip using the Connector.

I take that back. broadcastToSubscribers is not used to actually transmit the payload, just to notify that there's a payload available. Payloads are fetched using executeAction(). I do not know if there's a limit on the size of the return object for those calls.

One thing to try - reload the Connector plugin using "Reload in Interactive Shell", and watch that terminal window for crashes or error messages. Do the same with your plugin.

Thanks Joe, the paho library is what I used to test the external subscribe and was able to process the message and save the image. I will try the reload in interactive shell and see if I can see any error messages.

Darryl

On other thing to try - comment out line 323 ( device.updateStatesOnServer(stateList)) in your copy of plugin.py in the Connector plugin. If it works, then we know it's an issue with the size of the state value.

Question - when you turn on debug logging in the Connector plugin, and you get one of these MQTT messages, what do you get for this debug line:

Code: Select all

   MQTT Connector Debug            Indigo MQTT Broker: processReceivedMessage: test/foo/bar, payload: This is a test


No debug log entries generated for the image message topic.

Below is the related message that does show up in the debug log which has the timestamp and image type attributes for the image.

Code: Select all

   MQTT Connector Debug            MQTT Client: processReceivedMessage: ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/attributes, payload: {"timestamp":1705798126,"type":"interval"}
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client) Test: type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client) Test: Matching complete, is_match = False


The message topic is the same as above except instead of "attributes" topic, it should show "image" with payload of the binary image.

I'll try commenting out the line you suggested and see what happens.

Hmm. I'm thinking that large message is failing processing even before it gets to that point. The code assumes that the payload is encoded text. It just can't handle binary data. And there's no way to know at that point what the payload data type is. There appears to be a field in the raw MQTT protocol that would indicate the type, but I don't see any way to access that using the Paho library.

Commented out line 323 didn't change anything, I agree it looks like the large message is failing even before it gets to that point. I posted my own image topic message with "Test" as the payload and everything processed.

Is the code "before" processMessage the Paho library? I used that library in my external test and it worked just fine. I installed it using:

Code: Select all

pip3 install paho-mqtt


pip3 show paho-mqtt has the following output:

Code: Select all

Name: paho-mqtt
Version: 1.6.1
Summary: MQTT version 5.0/3.1.1 client class
Home-page: http://eclipse.org/paho
Author: Roger Light
Author-email: roger@atchoo.org
License: Eclipse Public License v2.0 / Eclipse Distribution License v1.0
Location: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages
Requires:
Required-by:


Is it possible there was an issue with older versions of paho-mqtt? It looks like you're using paho 1.5.1.

Darryl

Nope, I downgraded paho-mqtt to version 1.5.1 and it still handled the binary image.

This is my test code:

Code: Select all

# python3.6

import random

from paho.mqtt import client as mqtt_client


broker = '192.168.1.10'
port = 1883
topic = "ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/image"
# Generate a Client ID with the subscribe prefix.
client_id = f'subscribe-{random.randint(0, 100)}'
# username = 'emqx'
# password = 'public'


def connect_mqtt() -> mqtt_client:
    def on_connect(client, userdata, flags, rc):
        if rc == 0:
            print("Connected to MQTT Broker!")
        else:
            print("Failed to connect, return code %d\n", rc)

    client = mqtt_client.Client(client_id)
    # client.username_pw_set(username, password)
    client.on_connect = on_connect
    client.connect(broker, port)
    return client


def subscribe(client: mqtt_client):
    def on_message(client, userdata, msg):
        print(f"Received a message from `{msg.topic}` topic")
        topic_parts = msg.topic.split("/")
        indigowebpath = "/Library/Application Support/Perceptive Automation/Indigo 2022.2/Web Assets/public/images/"
        indigosnapshotfilename = topic_parts[1] + '-' + topic_parts[3] + '-snapshot.jpg'
        test_file = open(f'{indigowebpath}{indigosnapshotfilename}', 'wb')
        test_file.write(msg.payload)
        test_file.close()

    client.subscribe(topic)
    client.on_message = on_message


def run():
    client = connect_mqtt()
    subscribe(client)
    client.loop_forever()


if __name__ == '__main__':
    run()

The problem is not the Paho library, and I'm sure your code works as intended. As I said before, the plugin was designed to handle TEXT payloads (raw or JSON), not binary data. Also, there's some code in there that isn't designed for really large payloads (like images).

I think I've revised the code to handle the binary data, at lease for your use case. I don't have any way to generate an MQTT message with a large binary payload. None of the tools I use can do it.

I'm just hoping that the code changes won't break existing users of the plugin.

Try https://github.com/FlyingDiver/Indigo-M ... g/2023.0.1

The good news is the image messages are appearing in the MQTT Connector debug log, bad news is that when I subscribe in my plugin, I get the following (see the three error lines):

Code: Select all

Jan 21, 2024 at 10:34:19 AM
   MQTT Connector Debug            MQTT Client: processReceivedMessage: ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/image
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Matching complete, is_match = True
   Trigger                         ringmqtt Trigger (MQTT Client)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): Matching complete, is_match = False
   MQTT Connector Debug            MQTT Client: processReceivedMessage: ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/attributes
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Matching complete, is_match = True
   Trigger                         ringmqtt Trigger (MQTT Client)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): Matching complete, is_match = False

Jan 21, 2024 at 10:34:46 AM
   Error                           XML Parse Error: not well-formed (invalid token)
   Error                           In client packet stream
   Error                           On character 248 of line number 1.
   MQTT Connector Debug            MQTT Client: processReceivedMessage: ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/image
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Matching complete, is_match = True
   Trigger                         ringmqtt Trigger (MQTT Client)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): Matching complete, is_match = False
   MQTT Connector Debug            MQTT Client: processReceivedMessage: ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/attributes
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Matching complete, is_match = True
   Trigger                         ringmqtt Trigger (MQTT Client)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): Matching complete, is_match = False


And then it looks like the subscribe process hangs and no further messages are processed until I restart.

I'll try to figure out how to generate a message with binary payload and get back to you.

Turn on Verbose Debugging and test that again. Not sure we'll get anything more, but maybe.

Doesn't look like it gave any more, but verbose did spit out the payload image (lots of data) below these messages. Here is the snippet around the Error lines:

Code: Select all

   Trigger                         ringmqtt Trigger (MQTT Client)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): topic_parts = ['ring', 'dpwhwp-3qqos-0', 'camera', '0cae7dc23ab7', 'snapshot', 'image']
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): match_list = match_list : (list)
     Item : Match: homeassistant (string)
     Item : Any:  (string)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): match.split = 'Match' 'homeassistant'
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): topic_part 0 = 'ring'
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): Match Failed: 'homeassistant' != 'ring'
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): Matching complete, is_match = False
   MQTT Connector Debug            MQTT Client: on_message msg.topic='ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/attributes', msg.payload=b'{"timestamp":1705854709,"type":"interval"}'
   MQTT Connector Debug            MQTT Client: processReceivedMessage: ring/dpwhwp-3qqos-0/camera/0cae7dc23ab7/snapshot/attributes
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): topic_parts = ['ring', 'dpwhwp-3qqos-0', 'camera', '0cae7dc23ab7', 'snapshot', 'attributes']
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): match_list = match_list : (list)
     Item : Match: ring (string)
     Item : Any:  (string)
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): match.split = 'Match' 'ring'
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): topic_part 0 = 'ring'
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Match OK: 'ring' == 'ring'
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): match.split = 'Any' ''
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): topic_part 1 = 'dpwhwp-3qqos-0'
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Skipping 'Any' component: 'dpwhwp-3qqos-0'
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Matching complete, is_match = True
   MQTT Connector Debug            MQTT Client: fetchQueuedMessageAction, queue = ##ring## (187)
   Error                           XML Parse Error: not well-formed (invalid token)
   Error                           In client packet stream
   Error                           On character 248 of line number 1.
   MQTT Connector Debug            MQTT Client: queueMessage, queue = ##ring## (188)
   MQTT Connector Warning          Queue for message type '##ring##' has 187 messages pending
   Trigger                         ringmqtt Trigger (MQTT Client)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): topic_parts = ['ring', 'dpwhwp-3qqos-0', 'camera', '0cae7dc23ab7', 'snapshot', 'attributes']
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): match_list = match_list : (list)