Introducing the Shims plugin

Posted on
Sat Aug 10, 2019 1:52 pm
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Introducing the Shims plugin

Not incredibly fond of the name, but it's what we were referring to them as, so I'm going with it for now. Open to suggestions.

So the holy grail (of the week) is a flexible way to present IoT devices using MQTT messages as Indigo devices. So that's what this plugin does. I don't promise that it will work with ALL MQTT payloads, but I think you can make it work with most and I'm looking for specific payloads that won't so I can do more with them.

This works hand-in-hand with the MQTT plugin. But the protocol used is deliberately designed so that it doesn't preclude the use of other plugins or scripts. Documentation is non-existent at the moment, so I'm going to do a walk-through on configuring a couple devices. This is a long post. Sorry, but don't try to skip around. Read it all the way through carefully.

Note: The most up-to-date version of these instructions is in the Wiki.

First, you need some IoT devices publishing messages, and an MQTT Broker. And the MQTT plugin. Get all of that working first. For this example, I'm going to use messages that publish a topic and payload like this example provided in another thread. In this payload, the device-specific name is the "wemos1" part of the topic.

Code: Select all
devices/wemos1/tele/SENSOR =
{
    "Time":"2019-08-04T09:16:14",
    "BME280":{
        "Temperature":84.3,
        "Humidity":31.3,
        "Pressure":995.74
    },
    "TempUnit":"F"
}


We have one device with three useful data items in the payload. We'll create THREE indigo devices, each representing one payload value. As a bonus, all three devices will have all three values as states.

First, make sure that the MQTT plugin Broker devices is subscribed to the topic:

Screen Shot 2019-08-10 at 3.00.55 PM.png
Screen Shot 2019-08-10 at 3.00.55 PM.png (50.48 KiB) Viewed 4436 times


You could do a more specific subscription, but then you would need one for each device.

Next, we need a trigger for this topic to tell the MQTT plugin to do something with it when it's received. I considered automatically creating a trigger for each subscribed topic, but it's not possible with the current API.

Screen Shot 2019-08-10 at 3.06.17 PM.png
Screen Shot 2019-08-10 at 3.06.17 PM.png (144.21 KiB) Viewed 4436 times


This is a "Topic Component Match" trigger. You need to pick the Broker device, then build a list of topic component match items. Each item in the list (going down) matches a component of the topic string (delimited by the slashes, left to right). So this matches the topic string for "wemos1", with a wild-card in the second spot. Which means it'll match "wemos1" or anything with an otherwise identical topic string. Again, it could have been designed to exactly match this one payload, but it's not required. To change the topic list, just delete components and add them back in. Unfortunately, I couldn't figure out a way to allow re-ordering of the list items.

At the bottom of this panel is a shortcut that performs an operation which would normally be done within the action list associated with this Trigger. It's here because 99% of the time you're going to want it, and because by doing this during trigger processing we're avoiding a race condition which could potentially change the data in the Broker device before the action would get performed. The last item here is the "Message Type". This is another field where I'm not sure what the best name is. It's a string that is a unique identifier for the kind of payload message this is. It's used to filter the messages obtained by the recipient device/script. Payloads with similar structures can use the same identifier. The Shims plugin is flexible enough that it doesn't require a unique identifier for every device. Just similar devices.

That's it for setup in MQTT. Next is to configure the Shims plugin. First release is available here: https://github.com/FlyingDiver/Indigo-S ... /tag/0.0.0

The devices we want to create to represent the wemos1 IoT devices are all Sensor devices with values (not just on/off). So create a Shims->Value Sensor Device.

Screen Shot 2019-08-10 at 3.32.51 PM.png
Screen Shot 2019-08-10 at 3.32.51 PM.png (162.35 KiB) Viewed 4436 times


Select the MQTT Broker device that's used for this IoT device. Enter the same Message Type you put in the Trigger. Now you need to tell the plugin where the unique identifier for this devices is, and where the state value is.

The unique identifier (name) of the device is in the second component of the payload, so make sure "Topic Component" is selected, and put a 1 in the field. I was being lazy when I coded this, so I didn't convert from 0-based array counting. :) Then put in the value that's going to be in that location that's unique for THIS device. Alternatively, the unique identifier could have been in the payload, so there's another option on the "Unique ID Location" popup. It works like we'll be doing in the next section. We're done with the Unique Identifier section.

Now we tell the plugin where the state value is. It could be in the topic (unlikely) or in the payload. Select "Payload Fields" for the "State Value Location" popup. Select type JSON. Now we get to a tricky field. Go back and look at the JSON payload for this topic. The data we want is in a nested dictionary. In Python, this would be "payload['BM280"]["Temperature"]. Since there are two dictionary keys involved, the "key" to reach this data is "BM280.Temperature". Yes, this will break if there are keys in the JSON data with periods in them. I did some research, and can't find any characters that aren't allowed, so I picked something that is unlikely. My second choice was ":". If this turns out to be a problem, I'll change this to allow specifying the delimiter you're using on a case by case basis.

Then pick the type of sensor. I've included all the ones that I did for the Masquerade plugin. If you need something else, just file an issue on the GitHub page. Please specify units and hopefully an existing state image to be used.

Lastly, we have a key to reach a dictionary of data to be presented as additional states for the device. In this case, specifying "BME280" tells the plugin to take all the entries in this dict in the payload and make them states in the devices.

And that's it. Save the device, enable the IoT device and both plugins, and let it rip....

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sat Aug 10, 2019 1:59 pm
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Introducing the Shims plugin

Another quick example for a device that doesn't use JSON in the payload. I think it's the Sonoff devices that report their state changes using a minimal raw payload. Something like:

Code: Select all
topic = devices/RelayDevice/update
payload = 'ON'

or

topic = devices/RelayDevice/update
payload = 'OFF'


Everything is the same up until you create the Shims device.

Screen Shot 2019-08-10 at 3.55.50 PM.png
Screen Shot 2019-08-10 at 3.55.50 PM.png (138.01 KiB) Viewed 4432 times


In this case, the payload type is Raw and the plugin looks for a payload with a simple string value in it. If the payload string is one of 'Off', 'OFF', or '0', the sensor is set to off. Otherwise on.

This part of the plugin will be enhanced. Sample payloads and how they should be handled would be appreciated.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sat Aug 10, 2019 8:20 pm
aaronlionsheep offline
Posts: 260
Joined: Feb 24, 2019
Location: Virginia, USA

Re: Introducing the Shims plugin

Just tested this out and it looks like works! Just a few things I noticed and a few changes that would be nice to see in a future version.

1) Your screenshots differ slightly from what MQTT v0.0.9 has. Your trigger screenshot shows the "Queue Message" checkbox and "Message Type" text field. However v0.0.9 doesn't have those 2 fields in the Event Settings. Those things are only present in the Trigger Actions when you select "Queue Last Message for Dispatch".

2) (Feature Request) My wemos device reports pressure in millibars and I would like to be able to convert it to inHg. In the Greensky plugin, I added a feature to create formulas where I could handle value conversion to change from different units. So for this case, I could type "x/33.8638" to do a simple unit conversion before storing the value.

3) (Feature Request) The sensors on my wemos device are not calibrated correctly (probably 90% of these cheap sensors will be incorrect out of the box). I know that the temperature sensor reports 12 degrees too high and the humidity sensor reports 9 percent too low. An offset field would be nice to see in order to do a rough calibration. Maybe this could be implemented using a formula (a simple x - 12 or x + 9).

4) (Bug) When selecting a Pressure Sensor Type, it looks like you have "Humidity", "Humidity (inHg)", and "Humidity (mb)". I think those last 2 are types and should be pressure devices since I have never seen humidity represented as inHg :shock:

Screen Shot 2019-08-10 at 9.54.37 PM.png
Screen Shot 2019-08-10 at 9.54.37 PM.png (15.49 KiB) Viewed 4386 times


I was able to break out the Wemos device into 3 separate indigo devices instead of using the "Weather Station" plugin device using the Greensky plugin. (Top 3 devices are broken out using the MQTT and Shims plugin - bottom device is the combo device using Greensky with formulas and offsets).

I was also able to get the temperature and humidity sensors into Homekit using HomeKit Bridge (no pressure since there wasn't;t a pressure sensor in HomeKit Bridge).

I like where these plugins are leading to!!
Last edited by aaronlionsheep on Sat Aug 10, 2019 8:32 pm, edited 1 time in total.

Posted on
Sat Aug 10, 2019 8:32 pm
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Introducing the Shims plugin

1. My bad. I forgot to post MQTT 0.0.10, which has support for that change. It's available now.

2 & 3. Would you put those in a GitHub issue. Adding conversions or corrections to the data in a very generic/flexible way will take some thought. It does seem to me that they're both asking for the same thing. If I put in the ability to do math on the data, you could do the conversion as well as the correction.

4. Yeah, typo in the devices.xml. Fixed for next release.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Aug 11, 2019 4:37 am
autolog offline
Posts: 3988
Joined: Sep 10, 2013
Location: West Sussex, UK [GMT aka UTC]

Re: Introducing the Shims plugin

Hi Joe,
Just tried this out with the Smappee payload, extracting data that isn't in the section with arrays.
It is working well. :)

In another post, you replied:
FlyingDiver wrote:
... Oh wait, no, the Smappee uses arrays of values in the payload. Can't help you with that (yet).

Looking forward to the yet update. :wink:

Can you add a type of Power to the list of Sensor Types?

Thinking about it and a suggestion; it might be better to specify a type of Sensor and the units to be used separately. This could then be combined with a math (offset and conversion) option discussed previously. Also maybe add in how many decimal places to show? If the units were a text field then the sensor value could be shown as both the native value e.g. 76.0 and UI value e.g. 76.0%

This is really a useful and stunning piece of work. Thanks for all your efforts. :D

Posted on
Sun Aug 11, 2019 5:18 am
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Introducing the Shims plugin

autolog wrote:
In another post, you replied:
FlyingDiver wrote:
... Oh wait, no, the Smappee uses arrays of values in the payload. Can't help you with that (yet).

Looking forward to the yet update. :wink:

Still pondering the UI for specifying how to parse the payload. Not simple.

autolog wrote:
Can you add a type of Power to the list of Sensor Types?

What units? Is there an appropriate device type for the UI?

autolog wrote:
Thinking about it and a suggestion; it might be better to specify a type of Sensor and the units to be used separately. This could then be combined with a math (offset and conversion) option discussed previously. Also maybe add in how many decimal places to show? If the units were a text field then the sensor value could be shown as both the native value e.g. 76.0 and UI value e.g. 76.0%


I'm trying to not make the UI any more complicated than necessary. It's already bad enough. I'll think about it.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Aug 11, 2019 6:00 am
autolog offline
Posts: 3988
Joined: Sep 10, 2013
Location: West Sussex, UK [GMT aka UTC]

Re: Introducing the Shims plugin

FlyingDiver wrote:
...
autolog wrote:
Can you add a type of Power to the list of Sensor Types?

What units? Is there an appropriate device type for the UI? ...

In the Indigo SDK, there is an Example Device - Energy Meter.indigoPlugin which shows how to display this.

Units = Watts. :)

Posted on
Sun Aug 11, 2019 6:23 am
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Introducing the Shims plugin

autolog wrote:
FlyingDiver wrote:
...
autolog wrote:
Can you add a type of Power to the list of Sensor Types?

What units? Is there an appropriate device type for the UI? ...

In the Indigo SDK, there is an Example Device - Energy Meter.indigoPlugin which shows how to display this.

Units = Watts. :)


Heck, you're a Brit. I thought maybe you wanted BTUs. :roll:

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Aug 11, 2019 9:35 am
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Introducing the Shims plugin

Release 0.0.1 - https://github.com/FlyingDiver/Indigo-S ... /tag/0.0.1

Key syntax for JSON payloads enhanced to support arrays.
Variable precision for ValueSensor devices
Adjustment function for ValueSensor devices
Power(W) sub-type added

Initial support for on/off device control. New Relay type device. Works with Sonoff.

I need examples of Topics/Payloads that control devices. All I have right now is a Sonoff/Tasmota device.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Aug 11, 2019 10:52 am
Colly offline
Posts: 535
Joined: Jan 16, 2016
Location: Ireland

Re: Introducing the Shims plugin

FlyingDiver wrote:
All I have right now is a Sonoff/Tasmota device.

Hi Joe, Planning to give MQTT a go over the next few days. You mentioned in an earlier post wrt Sonoff devices to watch this space (or something similar!) for the new plugin. I haven't yet powered up my device, so my question is - does it need the Tasmota update or is there another way around this?
Hat's off for all the development and effort in the MQTT area, it's somewhat of a game changer in my view as it appears to really open up some many more options with Indigo.

Posted on
Sun Aug 11, 2019 11:01 am
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Introducing the Shims plugin

Colly wrote:
FlyingDiver wrote:
All I have right now is a Sonoff/Tasmota device.

Hi Joe, Planning to give MQTT a go over the next few days. You mentioned in an earlier post wrt Sonoff devices to watch this space (or something similar!) for the new plugin. I haven't yet powered up my device, so my question is - does it need the Tasmota update or is there another way around this?
Hat's off for all the development and effort in the MQTT area, it's somewhat of a game changer in my view as it appears to really open up some many more options with Indigo.


I honestly don't know what it can do with the native firmware. I was pointed to the Sonoff in the context of using it with Tasmota for MQTT. I know that people who are working with the new plugins are also using Wemos and Smappee devices. Anything that does MQTT works.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Aug 11, 2019 12:39 pm
Colly offline
Posts: 535
Joined: Jan 16, 2016
Location: Ireland

Re: Introducing the Shims plugin

Thanks Joe - I'll give Tasmota a go.

Posted on
Sun Aug 11, 2019 6:32 pm
cuhouse offline
Posts: 144
Joined: Feb 21, 2007
Location: Virginia, USA

Re: Introducing the Shims plugin

Colly wrote:
FlyingDiver wrote:
All I have right now is a Sonoff/Tasmota device.

Hi Joe, Planning to give MQTT a go over the next few days. You mentioned in an earlier post wrt Sonoff devices to watch this space (or something similar!) for the new plugin. I haven't yet powered up my device, so my question is - does it need the Tasmota update or is there another way around this?
Hat's off for all the development and effort in the MQTT area, it's somewhat of a game changer in my view as it appears to really open up some many more options with Indigo.


The Sonoff devices have Itead’s firmware installed on them from factory. You have to utilize their EWeLink app. It doesn’t use MQTT which is where flashing with Tasmota comes into play.
I have read that another manufacturer (Shelly) has a device that will run with their app or can run MQTT. I am planning to order one to try with Flying Diver’s Plug-ins. They run around US $29. Much more expensive than Sonoff but the one I am looking at is UL listed.

Jody
( Loving these new MQTT plugins)

Indigo 2022.1.2, Big Sur v11.7.1, Dedicated late 2014 Mac Mini, PowerLinc 2413U.

Posted on
Mon Aug 12, 2019 6:15 am
FlyingDiver offline
User avatar
Posts: 7217
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Introducing the Shims plugin

autolog wrote:
In another post, you replied:
FlyingDiver wrote:
... Oh wait, no, the Smappee uses arrays of values in the payload. Can't help you with that (yet).

Looking forward to the yet update. :wink:


And I forgot the reply to this when I posted 0.0.1 yesterday. The new key syntax you want is channelPowers.[0].power. Done.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Mon Aug 12, 2019 6:42 am
AndyVirus offline
Posts: 257
Joined: Mar 18, 2014
Location: Newport Pagnell, UK

Re: Introducing the Shims plugin

I have created a Switch Device. It seems to work but not reliably. When i turn on the device, the payload is published to the correct topic and the status of the switch is changed to On. All is well except in the log it shows:

Shims Owl Lamp MQTT: Stopping Device
Shims Owl Lamp MQTT: Starting Device
Shims Owl Lamp MQTT: Stopping Device
Shims Owl Lamp MQTT: Starting Device

I then try to turn off the Switch Device and the action triggers but the switch status does not change. hit it a few more times and it does. This is not consistent and sometimes it works quickly but not always.

Trigger Sonoff Trigger
Trigger Sonoff Trigger
Trigger Sonoff Trigger
Shims Owl Lamp MQTT: Stopping Device
Shims Owl Lamp MQTT: Starting Device
Shims Owl Lamp MQTT: Stopping Device
Shims Owl Lamp MQTT: Starting Device

Should the Switch Device do the starting/stoping device twice every time i hit on or off?

Who is online

Users browsing this forum: No registered users and 3 guests