Recommendations on DIY topic/payload schemas

Hi all,
I went down a rabbit hole watching YouTube and making custom sensors/ controllers. I am planning to make more than a few, and I want to have an easily generalized, Indigo MQTT Connector schema.

I am wondering whether to put the device id for each esp32 in the topic or payload, and how to do this with the simplest Indigo device-creation and trigger-creation schema.

What are people finding to be the most scalable? I plan to have over 30 devices when all is said and done.

Edit: Some of these sensors may re-deploy to different rooms, so i am hesitant to have room names in the topic if i can avoid it.

I would do something like:

sensors/<sensor type>/<sensor id>/update

And then a well defined JSON payload.

Sent from my iPhone using Tapatalk

I would do something like:

sensors/<sensor type>/<sensor id>/update

And then a well defined JSON payload.

Sent from my iPhone using Tapatalk

Sweet, that was my inclination but i appreciate the confirmation!

[jay (support)]

In terms of the JSON payload, spend a little time really thinking about how to make it generally useful. For instance, we are discussing a relatively poor JSON in another thread. That design works, but requires extra/unnecessary processing when a better design would have made processing much simpler.

Post your proposed JSON here - I'm sure you'll get lots of feedback that will only improve it.

Thanks again for this- I am kinda over the moon about finally having and Indigo friendly and easily reproducible way to make and deploy all sorts of a awesome but affordable sensors. I hope to share to anybody else kludging their way around this.

In terms of the JSON payload, spend a little time really thinking about how to make it generally useful. For instance, we are discussing a relatively poor JSON in another thread. That design works, but requires extra/unnecessary processing when a better design would have made processing much simpler.

Post your proposed JSON here - I'm sure you'll get lots of feedback that will only improve it.

Just saw this, will do fo sho, Jay!

Edit: Wow- Just saw that thread, Jay. That’s crazy somebody got paid to implement that.

I would do something like:

sensors/<sensor type>/<sensor id>/update

And then a well defined JSON payload.

Just to expand on this a little. Having a specific sensor-type field in the topic makes it easy to have specific triggers for specific types of devices, which in turn allows for unique message-types. All of which make the pipeline to the Shims plugin more efficient.

I would do something like:

sensors/<sensor type>/<sensor id>/update

And then a well defined JSON payload.

Just to expand on this a little. Having a specific sensor-type field in the topic makes it easy to have specific triggers for specific types of devices, which in turn allows for unique message-types. All of which make the pipeline to the Shims plugin more efficient.

Shims is planned once i get these up, and that is why I am using your recommendation verbatim

Edit: I cannot wait to share my keg “remaining beers” monitor. That’s not a default Device type, no?

Alright, I am determined to make this work, hopefully with one clearly posted question (fingers crossed).
(I PROMISE I read your wiki article a bunch of times first)

Device: home-cooked esp32 and a bme280
Device publishes to: sensors/air/0001/update // 0001 is the device id
MQTT payload: {"id": 0001, "temperature": 80.6, "humidity": 47.6}

A trouble-shooting trigger in Indigo appears to capture these to variables alright:

indigo variables.png (83.92 KiB) Viewed 2070 times

The MQTT Connector runs of my Synology NAS, and has never seemed to be a problem. I THINK it is reporting correctly in MQTT Explorer:

mqttExplorer.png (69.97 KiB) Viewed 2070 times

MQTT Connector.png (133.46 KiB) Viewed 2070 times

The Trigger may be where I am having problems. I tried it with and without "End:" at the end of the "Topic Strings for Matching" box.

MQTT Trigger 1.png (193.7 KiB) Viewed 2070 times

MQTT Trigger 2.png (141.55 KiB) Viewed 2070 times

Here is the MQTT Shims Device:

MQTT shims 1.png (230.34 KiB) Viewed 2070 times

MQTT shims 2.png (324.92 KiB) Viewed 2070 times

And here are the errors I'm getting:

Code: Select all

Nov 11, 2020 at 5:28:02 PM
   MQTT Connector Debug            MQTT Connector Broker: Message received: sensors/air/0001/update, payload: {"id": 0001, "temperature": 80.6, "humidity": 47.7}
   Trigger                         MQTT espsensor trigger
   MQTT Shims Debug                received notification of MQTT message type ##espsensor## from MQTT Connector Broker
   MQTT Shims Debug                esp bme280 sensor: processMessages: '##espsensor##' sensors/air/0001/update -> {"id": 0001, "temperature": 80.6, "humidity": 47.7}
   MQTT Shims Debug                esp bme280 sensor: JSON decode error for state_location = payload, aborting


My main theory is that the payload needs to be some exotic byte array kinda thing, and I'm simply sending a string containing json and an '\n'. If you suspect that's the case, any easy way to change this in C?

Many thanks!

Take out the multi-states key (make it a blank field) and try again. Then if it works, put in ".", which represents the top level of the dictionary. That key has to be a single value, not a list.

I admit the note for that field isn't very clear. I should fix that. It's a plural because nested keys would have more than one term, like "results.data" or some such.

Thanks for the speedy response, but no luck.

The more I read, the more I realize that my sending json as a string of text just won’t fly; i suppose i have to serialize it, something like:

Code: Select all

StaticJsonDocument<256> doc;
char buffer[256] = whatever my string of json is
size_t n = serializeJson(doc, buffer);
client.publish("outTopic", buffer, n);


There may be a much easier way to serialize this.

Edit: having read up on the topic, it appears a string is about a serial as it gets. Maybe there is some weird character or something in my json?

Yeah, that payload kinda looks OK, but JSON is tricky about quotes and double-quotes and other stuff. I use a standard JSON library to convert to/from Python data structures. And the error you're getting is exactly the line that does the conversion from the JSON string to to Python.

So, I played around with your data in a JSON validator (https://jsonlint.com) and found the problem. It's this part:

Code: Select all

"id": 0001


The '0001' is not a valid number, per the JSON spec. You'll want to make that a string. The plugin is doing a string compare anyway, so it would have failed even if it was valid (1 is not the same as "0001").

I don't think you'll need to do anything else like the serialization step. My plugins are getting the JSON string just fine, it's just not valid.

BINGO! Thank you so much, Flying Diver, you nailed it.

A special thanks, I've been screwing around with DC electronics the last couple months, and it's absolutely awesome and satisfying to make something for $14 that does something and interfaces seamlessly with Indigo.

The BME will be a sensor for my wife's poison dart frog tank (it's a whol;e thing).
For me, my "what is the temp of my keg and how much beer is left in it" flow meter and temp sensor will be the next to have a Device in Indigo.

Thank you so much again, I'm really, really happy about all of the potential projects I can easily control and monitor now!!!

successmqtt2.png (41.21 KiB) Viewed 2026 times

(I am aware of the dubious name of the sensor. I have a running joke about how hard it is to put "amateur homemade" in front of a web search and not have it come back dirty)

The attachment successmqtt1.png is no longer available

Hopefully this is an easy question:

Everything is working pretty well as above, thoguh MQTTShims keeps throwing this error in the log:

Code: Select all

Nov 12, 2020 at 9:43:32 AM
   MQTT Connector Debug            MQTT Connector Broker: Message received: sensors/air/sensor0001/update, payload: {"id":"sensor0001","temperature":71.7,"humidity":51.3}
   Trigger                         MQTT espsensor trigger
   MQTT Shims Debug                received notification of MQTT message type ##espsensor## from MQTT Connector Broker
   MQTT Shims Debug                esp bme280 sensor: processMessages: '##espsensor##' sensors/air/sensor0001/update -> {"id":"sensor0001","temperature":71.7,"humidity":51.3}
   MQTT Shims Debug                esp bme280 sensor: shimValueSensor, key = temperature, data = {u'humidity': 51.3, u'id': u'sensor0001', u'temperature': 71.7}, value = 71.7
   MQTT Shims Debug                esp bme280 sensor: Updating state to 71.7
   Error                           device "esp bme280 sensor" state key sensorValue not defined (ignoring update request)
   Error                           device "esp bme280 sensor" state key sensorValue not defined (ignoring update request)

Nov 12, 2020 at 9:44:32 AM
   MQTT Connector Debug            MQTT Connector Broker: Message received: sensors/air/sensor0001/update, payload: {"id":"sensor0001","temperature":71.7,"humidity":51.2}
   Trigger                         MQTT espsensor trigger
   MQTT Shims Debug                received notification of MQTT message type ##espsensor## from MQTT Connector Broker
   MQTT Shims Debug                esp bme280 sensor: processMessages: '##espsensor##' sensors/air/sensor0001/update -> {"id":"sensor0001","temperature":71.7,"humidity":51.2}
   MQTT Shims Debug                esp bme280 sensor: shimValueSensor, key = temperature, data = {u'humidity': 51.2, u'id': u'sensor0001', u'temperature': 71.7}, value = 71.7
   MQTT Shims Debug                esp bme280 sensor: Updating state to 71.7
   Error                           device "esp bme280 sensor" state key sensorValue not defined (ignoring update request)
   Error                           device "esp bme280 sensor" state key sensorValue not defined (ignoring update request)


I suspect it is related to designating a value to go into the INdigo Device State, which shows a thermometer but no value.

Thoughts?

I also have the precision value set to 1, but the device states add a trailing zero- no idea i f this is related

Recreate that Shim device from scratch (don't duplicate the existing one). Use all the same values you have now. I've seen that error before, but it's usually from creating the device with a different device type, then changing it.