Formatters and Parsers

Now that base functionality of the plugin is complete (if not thoroughly tested), it's time to look at the data handling.

Published data needs to be formatted according to whatever schema the intended recipients are expecting, and incoming (subscription) data needs to be parsed from the sender schema into something useful for Indigo.

I need examples of each. I'd really like to come up with some sort of template system that's user extensible. To do that, I need as wide a variety of use cases as possible.

I am the person behind the JSON parsing and value manipulation for the Greensky MQTT plugin. I feel this is one of the most dynamic ways to pull data m an MQTT payload. It currently allows for simple JSON key/value movement, so it currently can’t handle pulling data from the nth element of a JSON array. The value manipulation actually gets run as Python dynamically. That allows for complex conversion such as converting a temperature value from Celsius to Fahrenheit. If the payload value was “230 V” and you only want the 230, then you could even simply do “x.split(‘ ‘)[0]”.

The limitation if the greensky plugin is that only one indigo device can subscribe to a topic. So the reason for the different types of greensky devices were so that I could extract temperature, humidity, and pressure all from one MQTT device that has one payload. It would be nice if your plugin would allow for say 3 independent indigo devices (one for each value you want to pull from a payload) and have each device pull different data from the payload.

Example:
This comes across over one topic.

Code: Select all

 indigo/wemos/tele/SENSOR = {"Time":"2019-08-04T09:16:14","BME280":{"Temperature":84.3,"Humidity":31.3,"Pressure":995.740},"TempUnit":"F"}

It would be nice to be able to pull the temperature, humidity, and pressure from this payload. That is why I created a “weather station” device for greensky because that plugin only lets one indigo device subscribe to a topic.

One last thing, I prefer to use ActiveMQ for the MQTT broker. I haven’t taken a look at your plugin so far, but it would be nice to be able to have the option to keep using ActiveMQ.

One last thing, I prefer to use ActiveMQ for the MQTT broker. I haven’t taken a look at your plugin so far, but it would be nice to be able to have the option to keep using ActiveMQ.

I don't see why it wouldn't. I've tested it with four different MQTT brokers (or at least four different servers running some version of an MQTT broker). Is there anything unusual about ActiveMQ?

One thing I think is an advantage over the GreenSky plugin is that this one will use TCP or WebSockets, can use SSL/TLS, and allows specifying the protocol (3.1 or 3.1.1).

I am the person behind the JSON parsing and value manipulation for the Greensky MQTT plugin. I feel this is one of the most dynamic ways to pull data m an MQTT payload. It currently allows for simple JSON key/value movement, so it currently can’t handle pulling data from the nth element of a JSON array. The value manipulation actually gets run as Python dynamically. That allows for complex conversion such as converting a temperature value from Celsius to Fahrenheit. If the payload value was “230 V” and you only want the 230, then you could even simply do “x.split(‘ ‘)[0]”.

Very interesting. I'll have to look into replicating that.

The limitation if the greensky plugin is that only one indigo device can subscribe to a topic. So the reason for the different types of greensky devices were so that I could extract temperature, humidity, and pressure all from one MQTT device that has one payload. It would be nice if your plugin would allow for say 3 independent indigo devices (one for each value you want to pull from a payload) and have each device pull different data from the payload.

Example:
This comes across over one topic.

Code: Select all

 indigo/wemos/tele/SENSOR = {"Time":"2019-08-04T09:16:14","BME280":{"Temperature":84.3,"Humidity":31.3,"Pressure":995.740},"TempUnit":"F"}

It would be nice to be able to pull the temperature, humidity, and pressure from this payload. That is why I created a “weather station” device for greensky because that plugin only lets one indigo device subscribe to a topic.

I haven't done anything on parsing topics directly into Indigo devices yet, but I'm thinking about it. When I get to that point, I'll be sure to keep this issue in mind.

This goes in the Formatters category. Release 0.0.5 is out - https://github.com/FlyingDiver/Indigo-M ... /tag/0.0.5

Added automatic publishing for Device changes
Added automatic publishing for Variable changes
Added template system for Device and Variable publishing

You can now specify devices and variables to automatically publish their updates. In the Broker device configuration, there are four panels for this capability. For devices, there's one panel to specify the devices which will have updates published. Then there's a second panel to specify a template for the formatting of the topic and payload fields. Then two more panels for variables.

The device/variable selection panels should be fairly self-explanatory. If not, let me know and I'll try to document sooner rather than later.

The Templates use the Mustache template system (http://mustache.github.io) using the Pystache library (https://github.com/defunkt/pystache). Default templates are included in all newly-created Broker devices, but I can't figure out how to get newlines and tabs into the defaultValue fields in the XML file, so the payload one isn't very readable. So here's some examples.

Say you want your payload to look like this (JSON):

Code: Select all

{
    'name': 'WeatherLink',
    'deviceId':'732253293',
    'model': 'WeatherLink Live',
    'address': '192.168.111.40'
    'states':
    {
        'did': '001D0A7107E9',
        'status': 'OK',
        'timestamp': 'Mon, 05 Aug 2019 14:46:51',
     },
}


Yes, I know I'm missing the versioning and payload-type identifier Jay suggested. It's just an example. To get this output, the template looks like this:

Code: Select all

{
    'name': '{{name}}',
    'deviceId':'{{deviceId}}',
    'model': '{{model}}',
    'address': '{{address}}',
    'states':
    {
    {{#states}}
        '{{name}}': '{{value}}',
    {{/states}}
     }
}


All the identifiers in the curly braces (they look sort of like mustaches, get it?) come from a python dictionary that the plugin creates that looks sort of like the one above. Not exactly, but close.

The #states and /states keywords are telling Mustache that this dictionary entry is actually a list of items, and this section should be repeated for each entry in the list.

If you want other data from the devices included in the dictionary so that you can get it into your payloads, just let me know.

The topic entry is pretty simple:

Code: Select all

indigo/devices/{{deviceId}}/update


The variable templates are similar:

Code: Select all

indigo/variables/{{variableId}}/update

{
    'name': '{{name}}',
    'variableId':'{{variableId}}',
    'value': '{{value}}'
}


Clear? Probably not.

Here's an interesting factoid - MQTT brokers don't care if there are multiple client devices at the same IP address. So you can create multiple Broker devices in the MQTT plugin all talking to the same MQTT Broker. Now why would you want to do that?

Say you have two groups of devices that publish MQTT topics. Different sets of sensors, from different manufacturers, that use different topic schemas. You could put the subscriptions in different client devices, and enable/disable each group separately.

Or you're sending the same Indigo data to two different collectors. One wants JSON, the other wants XML. Create two devices, and use different formatter templates for each.

The outbound templating looks really nice! It makes sense to ME as I am used to working with Jinja.

I think the most important piece for my situation is the inbound payload to Indigo device mapping. It seems very clunky to create a virtual device, 2 action groups, triggers, and then a state variable in order to pull data from each topic's payload. I also understand that it can be tricky to implement plugin devices that talk to indigo-defined devices.

Ideally for my situation, there would be plugin-defined devices for a relay (outbound only), boolean sensor (contact sensor, inbound only), value sensor (temperature, inbound only), and on/off device (light switch, inbound AND outbound).

I already have a lot of Sonoff MQTT devices (flashed with Tasmota) that have hard-coded payloads. The templating for the outbound payload will be a step close in order to be able to publish payloads in the expected format. I can get a flashed Sonoff device sent down to Florida if it helps!

I already have a lot of Sonoff MQTT devices (flashed with Tasmota) that have hard-coded payloads. The templating for the outbound payload will be a step close in order to be able to publish payloads in the expected format. I can get a flashed Sonoff device sent down to Florida if it helps!

That's a thought but I'm not averse to getting some myself and flashing them, assuming it doesn't take any special tools. If you could send me a PM with some sources for where to get them and links to the flashing tools, I'd appreciate it. No need to clutter up this thread with all that.

I think the most important piece for my situation is the inbound payload to Indigo device mapping. It seems very clunky to create a virtual device, 2 action groups, triggers, and then a state variable in order to pull data from each topic's payload. I also understand that it can be tricky to implement plugin devices that talk to indigo-defined devices.

Ideally for my situation, there would be plugin-defined devices for a relay (outbound only), boolean sensor (contact sensor, inbound only), value sensor (temperature, inbound only), and on/off device (light switch, inbound AND outbound).

I agree, and I've been having a side conversation on what it's going to take in Indigo to make this easier. But any changes there are going to be a ways out, so I'll probably go ahead and implement the shim devices in the plugin for now.

Looking for examples of topics and payloads coming from IoT devices that need to be mapped to virtual Indigo devices. I've got this one, which actually maps to three devices. Is any of this configurable in the client? I figure the first term in the topic must be. Are any others? Is "SENSOR" the name of the device?

Code: Select all

 indigo/wemos/tele/SENSOR = {"Time":"2019-08-04T09:16:14","BME280":{"Temperature":84.3,"Humidity":31.3,"Pressure":995.740},"TempUnit":"F"}


The name of the device is actually "wemos". I can only modify the "indigo/wemos" portion of the topic. "/tele/SENSOR" is one of the hardcoded pieces appended to what we can customize.

So with tasmota, we cannot modify the structure of the payload (which changes depending on the specific devices and which sensors you connect) and part of the topic is also hard-coded.

The name of the device is actually "wemos". I can only modify the "indigo/wemos" portion of the topic. "/tele/SENSOR" is one of the hardcoded pieces appended to what we can customize.

So with tasmota, we cannot modify the structure of the payload (which changes depending on the specific devices and which sensors you connect) and part of the topic is also hard-coded.

OK, so you can change the topic prefix.

The problem with the data as shown is that, as far as I can tell, there's nothing in either the topic or payload that's a unique identifier for that specific sensor.

So you could change it to:

Code: Select all

 indigo/wemosensor101/tele/SENSOR = {"Time":"2019-08-04T09:16:14","BME280":{"Temperature":84.3,"Humidity":31.3,"Pressure":995.740},"TempUnit":"F"}

Or similar? And you can configure each sensor individually?

BTW - I'm hoping you can PM me with info on getting some of those sensors and configuring them.

The name of the device is actually "wemos". I can only modify the "indigo/wemos" portion of the topic. "/tele/SENSOR" is one of the hardcoded pieces appended to what we can customize.

So with tasmota, we cannot modify the structure of the payload (which changes depending on the specific devices and which sensors you connect) and part of the topic is also hard-coded.

OK, so you can change the topic prefix.

The problem with the data as shown is that, as far as I can tell, there's nothing in either the topic or payload that's a unique identifier for that specific sensor.

So you could change it to:

Code: Select all

 indigo/wemosensor101/tele/SENSOR = {"Time":"2019-08-04T09:16:14","BME280":{"Temperature":84.3,"Humidity":31.3,"Pressure":995.740},"TempUnit":"F"}

Or similar? And you can configure each sensor individually?

BTW - I'm hoping you can PM me with info on getting some of those sensors and configuring them.

Yeah, so actually "wemos" was the unique identifier. This was a device that is just being used for testing, so it was simply named what it was.

So to control an air compressor purge valve (electronic valve) there is a Sonoff relay device that uses

Code: Select all

indigo/air_compressor_purge/stat/POWER

as a payload of "On" or "Off" is published to it from Indigo to automagically purge the air compressor every day for a few seconds.

Example of Smappee Power Monitor MQTT payload from topic servicelocation/UUIDofTheSmappee/realtime

Code: Select all

 servicelocation/UUIDofTheSmappee/realtime = {"totalPower":3683,"totalReactivePower":751,"totalExportEnergy":0,"totalImportEnergy":1535076699,"monitorStatus":0,"utcTimeStamp":1565171048482,"channelPowers":[{"ctInput":0,"power":3683,"exportEnergy":0,"importEnergy":1535076699,"phaseId":0,"current":162}],"voltages":[{"voltage":232,"phaseId":0}]}

Weber iGrill2 Temp Probes from topic bbq/probe1 (1,2,3,4) are just string value of the temp such as

250 or 34 (any value)

(Weber iGrill Bluetooth MQTT bridge: https://github.com/bjoernhoefer/igrill

Sonoff topic stat/sonoff_owl_lamp/POWER to view status. same to publish command to cmnd/sonoff_owl_lamp/power

On or Off

If that is of any use...Just the ones I currently use. There is a way to get my Bosch Dishwasher to talk MQTT but not figured out how yet

Example of Smappee Power Monitor MQTT payload from topic servicelocation/UUIDofTheSmappee/realtime

{"totalPower":3683,"totalReactivePower":751,"totalExportEnergy":0,"totalImportEnergy":1535076699,"monitorStatus":0,"utcTimeStamp":1565171048482,"channelPowers":[{"ctInput":0,"power":3683,"exportEnergy":0,"importEnergy":1535076699,"phaseId":0,"current":162}],"voltages":[{"voltage":232,"phaseId":0}]} ...

So how exactly do you get the Smappee to give you realtime data?