Trying to understand Templates with Shims (for DB output)

Hi. First, thanks for MQTT + Shims. Works, so far, without issues.
My issue is trying to get devices "normalized" within Indigo. I have three different types of temperature sensors, each arriving via a different plugin (NOAA, native Indigo Aeotec Z-Wave, MQTT/Shims). As might be expected, each is represented somewhat differently in the Indigo state database. My goal is to try and normalize them enough so that as they are being pushed to a database for graphing that each type of device doesn't have to be individually known. (This actually falls into a general Indigo defect: they need to have some abstract device types so that developers have guidance on representing things in the internal database. But they don't, so, OK, here we go...)

So, I found the concept of "templates" in MQTT, and my question is "how can I build templates that control which values go into which fields?" For example, I would like to specify a string for "subModel" for my shim'ed device.
Similarly, although the shim GUI gives me the option to describe something as "Temperature (F)," it puts the data into sensorValue and not, for example, state.sensorValue state.temperatureF (like the NOAA plugin does). I'm actually not sure why it's asking what type of data I'm providing, actually.

I guess what I'm asking for is just a few words on how Templates work. Can I create arbitrary data elements and make them static? Can I send values from the MQTT->Shim-> particular values in the data structure?

With a little boost, I think I can figure all this out. I can fix it later "in the mix" (by handling things in the database), but if I can normalize to make a Shim'ed device look more like an Aeotec sensor (code written by Indigo-land!) then things will be simpler going in.

As data scientists always say, "data cleaning early saves headaches later."

Thanks again all...

There's two types of templates in the Shims plugin, and neither does what you want.

The first type is for defining how to format the payload for Indigo devices you're publishing to the MQTT broker, presumably to use in some other MQTT client.

The second is for describing a Shim device for a specific MQTT topic/payload (usually some specific IoT device) so that it's easier to create new instances of that device later. The intent is for people to send me those templates so that I can include them in plugin updates. Not too many people have done that.

Similarly, although the shim GUI gives me the option to describe something as "Temperature (F)," it puts the data into sensorValue and not, for example, state.sensorValue state.temperatureF (like the NOAA plugin does). I'm actually not sure why it's asking what type of data I'm providing, actually.

That selection is purely for the Indigo UI, control pages, etc. It sets the UI value of the device. The actual device value is the pure number, without any units on it. Which is what you should be storing in the DB, IMO.

I'm not understanding what difference you're seeing between a Shim set up as a temperature sensor, and a Z-Wave multi-sensor's temperature sensor device. Both of them have a sensorValue of the temperature, as a number (no units).

Thanks for the reply & the guidance. Sounds like "templates is not what I want" in the sense that I am rolling my own input to MQTT (it's a Pi with 433 MHz sensors and an SDR that is publishing to MQTT) and so there's no application for others to use this.

As you know there's a ton of device attributes (not sure that's the word they use inside of Indigo: these are the properties that may or may not vary over time), and for the NOAA plugin, I get (just showing some examples):

Grafana Home Dashboard JSON representation of device Lind Road Temperature:
Grafana Home Dashboard deviceTypeId: station (INCLUDED)
Grafana Home Dashboard displayStateId: temperatureString (NOT INCLUDED)
Grafana Home Dashboard measurement: weather_changes (NOT INCLUDED)
Grafana Home Dashboard model: Weather Station (INCLUDED)
Grafana Home Dashboard subModel: (INCLUDED)
Grafana Home Dashboard state.temperatureF: 55.0 (NOT INCLUDED)
Grafana Home Dashboard state.temperatureF.num: 55.0 (NOT INCLUDED)

For the Aeotec:

Grafana Home Dashboard JSON representation of device Patio Sensor Temperature:
Grafana Home Dashboard measurement: device_changes (NOT INCLUDED)
Grafana Home Dashboard model: Multi Sensor 6 (ZW100) (INCLUDED)
Grafana Home Dashboard name: Patio Sensor Temperature (NOT INCLUDED)
Grafana Home Dashboard sensorValue: 60.4 (INCLUDED)
Grafana Home Dashboard subModel: Temperature (INCLUDED)

And for my Shim'ed MQTT device:

Grafana Home Dashboard JSON representation of device Temperature-Office:
Grafana Home Dashboard measurement: device_changes (NOT INCLUDED)
Grafana Home Dashboard model: MQTT Value Sensor Device (INCLUDED)
Grafana Home Dashboard sensorValue: 51.44 (INCLUDED)
Grafana Home Dashboard subModel: (INCLUDED)

My goal is to present the device to the Grafana/InfluxDB plugin in a way so that it "looks like" a weather temperature sensor, rather than a generic device, and then write the Grafana dashboard so that it pulls out these devices with minimum human intervention in the design of the graphs. I totally understand that this is "not your problem" in the sense that I am dealing with another plugin, but it does seem like it would be nice if i could control (template?) how the device is presented to Indigo so that it mimics a device more closely, for example by controlling the "subModel" or putting the value in state.TemperatureF rather than sensorValue.

I'm not saying I can't work around it; I'm just wondering if there's a way within the structure of the code you've got that will let me define the presentation of the shim'ed device to Indigo more precisely. My real fear is that 1, 2, 3 years down the road, you'll change YOUR mind on how the device is presented and then it's back to the drawing board to re-engineer the path and find out what changed and what I need to change in the dashboard.

Anyway, sounds like i just need to suck it up and deal with the devices that I get... but, if you're thinking about directions for templates and how to present MQTT Shim'ed devices as Indigo devices, then maybe consider this an idea.

Thanks! Joel

Well, hold on. It's not that bad. First, let's clear up some terminology. Things like "subModel" are properties of the device. state is a special kind of property, which is a dict of values, managed by the plugin. They're either defined in the Devices.xml file statically, or as the Shims plugin does, created dynamically.

I don't use subModel in Shims, but I could certainly add a text field to the device configuration panel so you could put anything you want in there.

As for getting the state "TemperatureF" to include the same value that's in sensorValue, you can do that in the payload you're generating on the rPi and sending via MQTT. You'd need to show me a sample of the MQTT payload you're sending now for me to provide any guidance.

You'll note that the sensorValue representation is identical for both the Aeotec device and the Shim device. The only real difference is the lack of the subModel field. Which is totally optional in Indigo and many (most?) plugin devices don't use it.

Also, NOAA is the odd one out here. It's a weather station, not a temperature sensor. Temperature is just one of the many attributes it makes available as a device state. That's why it's not using sensorValue. You could make it look like a temperature sensor using my Masquerade plugin.

OK, thanks again for the advice & guidance. Sounds like it's time to get deep into Influx and Grafana and pull things together...

I was thinking that the NOAA device approach was the cleanest; the Aeotec idea of creating N devices, one for every internal sensor, has always seemed a little funky to me. I guess it gets back to the need for a bit of standardization coming from Indigo on how to represent things in their product. Right now, the plugins overwhelm the core product so you end up with a cacophony of voices each with their own idea of how things might be represented, and it's hard to discern what's best. For most of us, it doesn't matter, but now that I'm trying to take the output of one plugin and stick it into the input of another, suddenly the disconnect hits me in the head a bit.

Just in case anyone finds this thread and cares, what I'm doing is taking inexpensive AcuRite 433 MHz weather sensors ($13/ea on Amazon) and pairing them with an older Pi that has a software-defined radio (SDR) on it ($25 on Amazon). The Pi has little more than a Mosquitto MQTT broker, something to read off the SDR, and a shell script that glues the two together: read a reading, post it to MQTT, lather rinse repeat.

I have control of the MQTT, so I'm just pushing /home/<probename>/temperature, /humidity, /battery to MQTT as raw; not even worth putting JSON into it. And of course the MQTT plugin pulls the variables out, and the Shim creates a virtual device to advertise the values.

Thanks again for taking the time to help! Joel