questions about Nest plugin, states in Grafana

I'm stumbling over some issues with Nest Thermostats and trying to display data from them using Grafana.

I realize my situation may be different from others - I personally have a few Nest thermostats and also have some other types of thermostats as well, and am often trying to use Grafana to graph their data in similar ways when possible.

First, I believe it is the case that Nest thermostats don't update the basic states from indigo.thermostat class like heatIsOn, coolIsOn, and fanIsOn. This makes it hard for me to do things like write a single query across multiple thermostats to plot state, or show the amount of runtime for heat/cool per hour or whatever. Seems like this should be fairly easy to fix (on the Nest thermostat plugin) and I will post an issue there. Anyone know any convenient workarounds or query magic I can use to paper over the discrepancy?

In trying to inspect and understand what is going on with this plugin and GHD's treatment of it, I'm poking around in Indigo.

I am looking at Nest thermostat devices in Indigo (looking at the device details window). This (if I understand correctly) lists the device's custom states in scrolling window at the bottom (which you don't see by default unless you scroll or stretch the bottom part of the Indigo window. It seems there is a daunting list of stuff there which comes from a) Nest is a fairly complex device with regular setpoints, eco set points, locked set point limits, and many other details. b) supports Fahrenheit and centigrade and has separate states for each. and c) seems to support decimal value and convenience rounded integer values for centigrade temperatures (presumably for user convenience when presenting information on a control page or similar) . d) there seem to be a mix of naming conventions, not always consistent - sometimes camelCase, sometimes_underscores, sometimesuniformcase, sometimes.dots, etc. No question that there are quite a lot of states to sort thru.

Currently, my Nest thermostats are in my list of included devices, so it seems that all of their states should be sent to InfluxDB. So I believe in the bottom part of the Indigo window I'm seeing the custom states for the device, meaning those states are specifically defined by the Nest Home plugin. Other "regular" (non-custom) device states (aka "properties"?) are shown generally either in the top of the window or in the case of some properties like lastUpdated, as a separate column in the device list view, or in a few cases (folderID) may not be viewable directly.

From an earlier inquiry on this subject, my understanding is that GHD creates field names from custom states by prepending "state." to the name to avoid possible collisions with properties of the same name. So if a I have a device which is an instance of a plugin that defines custom state "foo" (and I include this state via GHD's config), then I expect to see a corresponding field in InfluxDB called "state.foo". Properties of a device, on the other hand, are represented in InfluxDB with field names that match the property names (that is, without "state." prepended).

Also, it seems that when there is a state whose values are strings, but whose values can also be converted to a number (including strings that parse as numbers or can be interpreted as boolean values like "yes", "no" "true", "false", etc.), GHD creates an additional field with the same name with ".num" appended to the end. So for example, if I have a custom state called "isBar" and it takes on values "Yes" and "No", then GHD create a field with name "state.isBar" and an additional field with name "state.isBar.num" and in this field, it will store values converted to numerical values (presumably Yes->1.0, No->0.0).

Using Plugins>Grafana Home Dashboard>Explore Device, I'm poring over a bunch of information dumped to the logs, but still not quite able to understand what I'm seeing there. Some specific questions and issues:

• In Indigo, I can see custom state for my Nest thermostat devices: isheating =Yes. I don't see the corresponding fields I would expect to see (state.isheating and maybe state.isheating.num) listed in the json dump generated by GHD, nor does it exist in influxDB. Why is this?
• In the json dump, "name" and "folderId" appear. These I believe are tag keys (not fields) in influxDB. Also, I notice that "folder" does not appear - why, since it presumably has the same status?
• In the json dump, I notice both "folderId" and "folderId.num are present - is it expected/desired that these both exist? Is one a tag and one a field? Is there lost efficiency by having the tag ("folderId") be a string instead of a number - does influxDB even care about this?
• Basically same as previous question for "Id" and "Id.num".

As an aside... It could greatly simplify user comprehensibility of the "explore device..." dumps produced by GHD if a) a state were listed on a line with a simple annotation when the ".num" state is also included (this would cut the number of lines almost in half and greatly reduce unnecessary entropy in the output). b) the states were sorted by name, either simply by the actual field name or (bonus!) by the "root" name, so that for example fields state.humidity, humidifierIsOn, and state.humidityInput1 would come out adjacent in the sort. c) maybe either eliminate "special" fields/states from the output or flag them as such, since things like "name", "folder", "folderId" and friends are not things the user can really include/exclude via settings.

First, I believe it is the case that Nest thermostats don't update the basic states from indigo.thermostat class like heatIsOn, coolIsOn, and fanIsOn. This makes it hard for me to do things like write a single query across multiple thermostats to plot state, or show the amount of runtime for heat/cool per hour or whatever. Seems like this should be fairly easy to fix (on the Nest thermostat plugin) and I will post an issue there. Anyone know any convenient workarounds or query magic I can use to paper over the discrepancy?

I've had success with the "state.hvac_state". It's working for both heat and cooling for me.

Currently, my Nest thermostats are in my list of included devices, so it seems that all of their states should be sent to InfluxDB

I would be careful with this approach. Likely you are sending more data to Influx than needed. I'm very cautious with what I put in the "Include Device" list, espeically third party plugin devices. I prefer to manually include states.

GHD creates an additional field with the same name with ".num" appended to the end.

I'd have to dig into the code more, but in my experience you are correct. The original Influx plugin author, who I inherited the Influx client code from, does some tests on the property and will send both the string value and the numeric value to Influx, and will inform Influx of the types. This makes it easier to write some queries. To be honest, I use the .num or non-.num interchangeably in my metrics. Also, these ".num" states are kind of virtual states. The explore devices and explore states should show them, but they dont actually exist in the devices.

In Indigo, I can see custom state for my Nest thermostat devices: isheating =Yes. I don't see the corresponding fields I would expect to see (state.isheating and maybe state.isheating.num) listed in the json dump generated by GHD, nor does it exist in influxDB. Why is this?

This was a bug it seems I introduced in 1.0.7 -> 1.0.9. It only affects the Explore Device output. It should not affect how the plugin is logging the device data. Fixing it in 1.0.10, thanks for reporting.

In the json dump, "name" and "folderId" appear. These I believe are tag keys (not fields) in influxDB. Also, I notice that "folder" does not appear - why, since it presumably has the same status?

I'm honestly not sure about the folder stuff. I dont use it myself. A quick scan of the Influx client code that I inherited I do not see anything going on with folder, but the name and id are sent every time to Influx as special properties.

Another related question:

I had set up some graphs to show state "state.hvacOperationMode.ui" which seemed to work well and took on values "Heat", "Cool", "HeatCool", "Off") for a couple of my thermostats and this state somehow seems to have disappeared and my plots are broken. Ditto for "state.hvacFanMode.ui" with values Auto-on and On.

These states seem to have disappeared sometime in the last few weeks.

I had starting using these states without really thinking about it, but now I'm guessing that the ".ui" states were auto-created by the plugin (similar to the ".num" states) to handle standard enumerations in the Indigo Object Model for indigo.kHvacMode and indigo.kFanMode.

Was there a change made to the plugin sometime in the last couple of versions that removed these? Was it intentional?

I've posted 1.0.10. Update to that and see if this resolves these issues.

i've upgraded to 1.0.10. first thing i notice is that my discrete panels that referenced "*.ui" states seem to be back in good order. thanks!

For my nest thermostat devices, I use a discrete panel that is using state.heatOrCool, which takes on values "Heating", "Cooling", and "Saving Energy".

Sometime this afternoon, I notice that my plots covering my nest thermostat state info went a little weird. I'm not sure whether this related to the recent change to the GHD plugin I just installed today or something else.

I also see that the nest thermostat changed mode to "all off". I suspect this may be my thermostat going into Eco mode (I was out of the house for a few hours) and this is just how the (currently somewhat broken) Nest plugin records Eco mode? I don't use "Eco" mode that often, so I'm not very familiar with how it behaves.

In the attached graphic, you can see the action starts around 15:45. System goes into "all off" mode and state.heatOrCool seemingly repeats the same value "Saving Energy". In Grafana, each interval lasts 60 seconds and the intervals are 1 second apart from each other. So looks like this might be a small bug in how GHD is handling state updates, perhaps?

When I started this post, I thought this was going to be a 1.0.10 bug report, but on inspection, I see the same behavior in earlier situations where (I think) Eco mode kicked in.

I didn't have state.heatOrCool added to my configuration and therefore I cannot graph it historically. I've added it now and I'll monitor the behavior.

Just guesses, but you could turn on your Legend to see if the Discrete panel is considering those transitions to be different values. Check the transitions count. There may be differences in the text or very quick blips in changes to the state in between that are hard to see. Try zooming in to graph just a few minutes of data. You can use the Mappings tab to map multiple items together.

All of the changes since 1.0.7 have been as result of the thread you started and participated in about the volume of data being sent to Influx. I added code that reviews if the state value is a string. If it is, I compare it to the previous value before the "update event" and ignore case sensitivity. If there has not been a change, I do not send it to Influx. This reduced the amount of data going to influx substantially as I discovered that many of the previous value comparisons were not working properly for strings.

During that change, I also introduced a bugs where the "minimum update frequency" stopped working correctly for strings, and broke the Explore Device. Both of those are fixed in .10.

I mention this to explain what likely differences in behaviour you should expect -- reduced number of state updates to Influx for string values only.

Using the ability of the Discrete panel to show details about those little blocks shown in the graphic I attached, I observe:

Nest and GHD behave normally/correctly until (apparently) Eco mode kicks in. I don't see any of this behavior any other time. Also, Nest plugin has a known weakness that it was coded prior to when Nest introduced Eco mode (when they used home/away), so the oddities I'm observing in GHD output may result from the Nest plugin doing something odd during that interval.

The little blocks have start and end times like this:

Code: Select all

start      end        duration      separation from next
15:42:30   15:49:49     7:19           :00           < - preceding "normal" interval
15:49:39   15:50:40    1:01            :00           < - first of a long series of short odd/redundant intervals
15:49:40   15:50:41     1:01           :00           
15:50:41   15:51:42     1:01           :00   
....[ long sequence of  repeating intervals with duration of either 60 or 61 seconds and separation from the next interval of either 0 or 1 second - all with identical value "Saving Energy"]
16:53:57   16:54:58     1:01           :00
16:54:58   17:16:27     21:29        :00    < - heating resumes - state.HeatOrCool becomes "Heating"

For all the repeating short blocks, the value I see appears identical (Saving Energy) including capitalization. There's a possibility that there could be some invisible difference like extra spaces at the end or something that I can't see.

The times that I reported are presented values I see on the screen and are rounded to the second. So in cases where I show a separation of :00 between blocks in the Discrete graph, it is possible that what is actually happening is something like: the value of state.HeatOrCool is "Saving Energy" for 61 seconds, then state.heatOrCool has no value for a fraction of a second, then the value goes back to "Saving Energy" again, etc.

A wild guess is that this is somehow caused by the Nest plugin is getting a value for mode that it doesn't think is valid (Eco) and continually re-requesting the mode and then continually setting the mode back to "Saving Energy" - perhaps as a default/fallback value or perhaps because that was the last known valid state.

Still, looking from the outside, I have the question of why GHD is representing all those little blocks as though they were transitions, when the state seemingly remains the same on both sides.

I'll keep monitoring my own and see what I find. Though, I'm quite confident that the Discrete panel is able to successfully ignore repetitive values from Influx without showing as a transition. Prior to 1.0.7 the string comparison "bug" was sending unnecessary updates to Influx, and no one reported problems like this. It may be characters like you said. We'll figure it out.

about 4 hours of data so far, and no issues (rows 3 and 4 are heatOrCool). See my panel. Does your state.hvac_state and state.heatOrCool align with eachother?

FWIW, I scanned the code for Nest Home plugin and there's only one occurrence of the string "Saving Energy" in the code that I can find. So nothing so trivial to fix as accidentally having two different spellings, spacings, or capitalizations.

I've reproduced the behavior you are seeing with the rapidly changing states. I don't know what's causing the NEST plugin to do those changes, but GHD is working properly by reporting the states. The plugin is setting the custom state to both "Saving Energy" and "Off" . It looks like it is fighting itself. I would venture to bet this is also a plugin inefficiency for those of us running the plugin.

I guess my recommendation is to use the state.hvac_state rather than state.heatOrCool. And remove state.heatOrCool from the GHD config. They seem to serve the same purpose, but instead for hvac_state, saving energy = off.