Message Size Limitation for MQTT Connector?

Here is some code to generate a binary message if that helps, just need to change the image path to some image file on your system:

Code: Select all

# python3.6

import random
import time

from paho.mqtt import client as mqtt_client


broker = '192.168.1.10'
port = 1883
topic = "test/image"
# Generate a Client ID with the subscribe prefix.
client_id = f'subscribe-{random.randint(0, 100)}'
# username = 'emqx'
# password = 'public'


def connect_mqtt() -> mqtt_client:
    def on_connect(client, userdata, flags, rc):
        if rc == 0:
            print("Connected to MQTT Broker!")
        else:
            print("Failed to connect, return code %d\n", rc)

    client = mqtt_client.Client(client_id)
    # client.username_pw_set(username, password)
    client.on_connect = on_connect
    client.connect(broker, port)
    return client


def publish(client):
    msg_count = 1
    while True:
        time.sleep(1)
        #msg = f"messages: {msg_count}"
        f = open("/Users/darrylscott/Pictures/testimage.jpg", "rb")
        imagestring = f.read()
        byteArray = bytes(imagestring)
        result = client.publish(topic, byteArray, 0)
        #result = client.publish(topic, msg)
        # result: [0, 1]
        status = result[0]
        if status == 0:
            print(f"Send image to topic `{topic}`")
        else:
            print(f"Failed to send message to topic {topic}")
        msg_count += 1
        if msg_count > 2:
            break


def run():
    client = connect_mqtt()
    client.loop_start()
    publish(client)
    client.loop_stop()


if __name__ == '__main__':
    run()

The line I wanted to see was:

Code: Select all

   MQTT Connector Debug            MQTT Client: fetchQueuedMessageAction, queue = ##ring## (187)


Which tells me that this is actually happening when your plugin is fetching the mqtt message, not when it's being initially processed.

So my working theory is that the data in the Python dict being returned to your plugin is being converted to/from XML during the relay through the Indigo server, and the XML conversion can't deal with binary data in the dict. I'm trying to confirm this with the Indigo developers.

So what I think I'm going to have to do is an an option to the fetchQueuedMessage() call to ask the Connector plugin to encode the payload into something that XML can handle. Probably Base64. So you would need to be prepared to decode it on your end.

Two other comments -

1. The "# Python 3.6" line on your scripts is misleading. It's not actually doing anything, and you should be using Python 3.10 for everything having to do with Indigo, as that's what Indigo is using.

2. Please use the CODE tags in the forum editing bar when posting code or log output.

Thanks for the insight and tips Joe, still new to a bunch of this....really appreciate your help.

Darryl

Try this version: https://github.com/FlyingDiver/Indigo-M ... g/2023.0.2

This version takes an additional parameter in the call to fetchQueuedMessage. See below. Also see the line to decode the encoded payload after it's been fetched.

Code: Select all

import base64

mqttPlugin = indigo.server.getPlugin("com.flyingdiver.indigoplugin.mqtt")
if mqttPlugin.isEnabled():
    props = {
        'message_type':"#Test#",
        'message_encode': True
    }
    while True:
        message_data = mqttPlugin.executeAction("fetchQueuedMessage", deviceId=775702409, props=props, waitUntilDone=True)
        if not message_data:
            indigo.server.log("Queue Fetch - no data")
            return

        indigo.server.log(f"Queue Fetch, {dict(message_data)=}")
        indigo.server.log(f"Queue Fetch, {base64.b64decode(message_data['payload'])=}")


Tried the new version, still getting the XML error on the image message, however it's not hanging the process anymore as other messages are processing before and after the image ones.

I snipped the tail end of the image debug lines, then the error and then the next related attributes message.

Code: Select all

6\xe1\xf5\xd5D\xcc\x18\xf4\x8d\xe6\xad\xd7\xf7\xf25\x13\xc4\xbe\xea\x00\x93=h\x80\xdb\xbf\x95\x05x\xb7\xbf\xf4\x15v\xf0\xb7\xba\xb0\xd0\x8eCy{\xf8\x0f\xb2\xab\xc3\xefG\xa6\xff\x00ePx?\xc2k[\x8a\xd3\n8A\xb4H\xe3Zg\xcc\x1fQC\x1e\x13Z<4\xc6\x1a@<H\xfd\xfeT;\x8a\x91\xbe\xde^\x95s\xc2\x82\x7f\xbd\x00\x18\xb26\xfb\x8f\x94\xff\x00j\xa3<\xc0\x03\xaf\xba\x85\xf7j\xc3\x8f\xef\xe7I \x1e\xa0\xdbz\xda\xb0\xe1T\xa5\x89\xac\x93[5Z\xa9\xa7\x14\xbd\xd42\xd5\x943Zc0\xb5R\xfa\xadR\x80\t}6\xc9\x92\x89L\xf2\xd6\x1a$\xe7\xc8n\x8a\xe2\xb5+nG\x1eL\xc3\xe0\x7f\xb4W_\x9b\xc6k\x94\xd6\xff\x00;/\xfe\xa3~\x95VO\x84t0\x15z\xa8\xabTQ\x8c\x9a\xc9\xa9YA\xa6\xd45*V02\x964>\xd0:_\xc3\xc8/\xc4O\xcd=W\xd3\xcc|\xe9\x1e\xb4\xf0\xf8\xfd\x94\xc8\xc3\xbe}F\x9dT=\xca\x14\xc41>~U\xce\xea\xfd\xa8^W\x0e\xc3\xab\xf5\xff\x00\x08\xe9\xef;\xd0\xb5_\xf0\x18\x7f\xc1\xf6\nD\xa9:+\x02\x16&\x89\x89C0\x9e\x1dh4\xe3\x0f\x8a\xa3\xb7l\xc6uX\xa9\xf2\xd3,T\xfdjdU";\x08+k\x05Z\x9c\x06Z\x9c\xbc\xb5\xf5;/\xbf\xd7\xd0q4\x97\x8cK\x96\x9f\xdc\xf9\xfd\xbf1N\xb5\xdf\xf2\xfd\xe7\xfd\x13M\xb0p>\xf1\xfe\x8a\xd6\xc9\xd2K\xa8\x0f"\xb4\x92;W\xc5\x1b\x9e\x89\xfb|\xaa\xd5Q\xfc\xdc\xbf/\xb2\xa8\xc8\xe9\x16cV\xccf\\B\xd1\xb9=<\xcf\x99\xf4\xa4\x8c\x84\x92G\xc7\xd4\xff\x00oJ}\x9f\xf9\xab\xfd-\xfaR{q\xa8\xcb\xa9|\x9d\x02c&\xa91[Tn\x142\xa6\x01\x9aj\x9d*\xb5\x7f\xbbJ9\xb8\xc1,\x07\x9d/\xc5\xa2)\x0f\x07\xf3W\xdfK\x86\xa8\x96\xab\xcc\x95\x8b\xbf\x90*\x03\x98\xa3\xd3l\x94\xe8g\xa0\xd1\x8e\xf4\xd9\x8e\xf4v\xe3M\x9b\x8dXGg\xff\xd9'}
   MQTT Connector Debug            ringmqtt Trigger (MQTT Client): Matching complete, is_match = True
   Trigger                         ringmqtt Trigger (MQTT Client)
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): type = topicMatch, broker = 790754084
   MQTT Connector Debug            ringmqtt-HA Trigger (MQTT Client): Matching complete, is_match = False
   Error                           XML Parse Error: not well-formed (invalid token)
   Error                           In client packet stream
   Error                           On character 481 of line number 1.
   MQTT Connector Debug            MQTT Client: fetchQueuedMessageAction, dict(action.props)={'message_encode': True, 'message_type': '##ring##'}
   MQTT Connector Debug            MQTT Client: message={'version': 0, 'message_type': '##ring##', 'topic_parts': ['ring', 'dpwhwp-3qqos-0', 'camera', '0cae7dc23ab7', 'snapshot', 'attributes'], 'payload': b'{"timestamp":1705868871,"type":"interval"}'}
   MQTT Connector Debug            MQTT Client: fetchQueuedMessageAction, dict(action.props)={'message_encode': True, 'message_type': '##ring##'}
   MQTT Connector Debug            MQTT Client: fetchQueuedMessageAction, dict(action.props)={'message_encode': True, 'message_type': '##ring##'}



Darryl

Did you updates the props as in the example?

Code: Select all

    props = {
        'message_type':"#Test#",
        'message_encode': True
    }


Yes, here is the code I used:

Code: Select all

 
       props = {'message_type': RINGMQTT_MESSAGE_TYPE, 'message_encode': True}
        brokerID = int(notification['brokerID'])
        while True:
            message_data = self.mqttPlugin.executeAction("fetchQueuedMessage", deviceId=brokerID, props=props, waitUntilDone=True)
            if message_data is None:
                break
            #self.logger.debug(f"processMessage: {message_data}")

            topic_parts = message_data["topic_parts"]
            payload = base64.b64decode(message_data['payload'])


Hmm. That error is coming before the fetchQueuedMessage. So it's not happening on the fetch.

The debugging log was so large, it was getting confusing where the Error was getting generated, but I think this made it clearer.

Code: Select all

Jan 21, 2024 at 7:04:22 PM
   Trigger                         ringmqtt Trigger (MQTT Client)
   Ring MQTT Warning               About to call fetchQueuedMessage
   Trigger                         ringmqtt Trigger (MQTT Client)
   Ring MQTT Warning               About to process camera snapshot image message
   Error                           XML Parse Error: not well-formed (invalid token)
   Error                           In client packet stream
   Error                           On character 481 of line number 1.
   Ring MQTT Warning               About to call fetchQueuedMessage
   Ring MQTT Warning               About to process camera snapshot attributes message
   Ring MQTT Warning               About to call fetchQueuedMessage
   Ring MQTT Warning               About to call fetchQueuedMessage



So it looks to me like it's when I go to update the indigo device state and/or write out the file, this is the code right after "Ring MQTT Warning About to process camera snapshot image message"

Code: Select all

            device.updateStateOnServer(key="snapshot_image", value=payload)
            test_file = open('/Users/darrylscott/Pictures/testimage.jpg','wb')
            test_file.write(payload)
            test_file.close()


So I commented out the following line and the error is now gone:

Code: Select all

            device.updateStateOnServer(key="snapshot_image", value=payload)


And the externalized file is storing the image correctly.

Are you still updating the states with image? Should I have been able to update the states or is that the issue you're asking the indigo developers about?

I don't think I need to update the states with image, what I needed was to be able to store the snapshot in place that where I could using a refreshing url on a control page.

Darryl

May have answered my own question, I need to base64 encode to store it in a state and decode when I want to use it, correct?

Darryl

Yes, you can't put binary images in a state. But I would try very hard to avoid doing that anyway. That's a huge amount of data to store that way. You're better off writing it to file.

Oh, you are writing it to a file. So why put it in a state? if anything, I would put the path to the file in a state.

Yep, that's the new plan, I just updated the code to put the path in the state and only store the image in a file.

Thanks again for all your help.

Darryl