Enabling usage tracking

Posted on
Wed Aug 30, 2017 5:46 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Enabling usage tracking

(this discussion is continued from this topic regarding extending Z-Wave range)

After reading @MartinG's post, jay (support) wrote:
The code in the plugin that populates usage data is commented out. I can't remember exactly why, but I believe it's because the library used as the basis for the plugin didn't handle it correctly (but I could be misremembering).

The source is available in case someone wants to try it out - uncomment lines 126-128, 140-142, and 317-320 in the plugin.py file. I think you'd need to recreate any devices to have the changes picked up.

It might work, but it also might not. ;)
after which, I wrote:
...Well, I guess it could work. I uncommented the lines you noted and do not see the usage data. But, with debugging turned on, I see that the plugin is getting the data. So, the question is why it doesn't display. BTW, the plugin also does not update on changes in brightness either. I'll look into it later this week. Thanks for the pointer...

I have found the source of the problem noted above. It was in processUpdate where
    keyValueList.append was used
      keyValueList.append({'key':'energyCurLevel', 'value':int(deviceInfo["watts"])})
    instead of updateStateOnServer
      dev.updateStateOnServer("curEnergyLevel", value=deviceInfo["watts"])
The same error was found in the code to update the brightness level and kwh readings. This will probably clear up a bunch of other minor state display issues. However, there still appear to be problems. A number of properties are declared that don't seem to ever be used and which duplicate device states. For example, there is a property energyCurLevel that duplicates the state curEnergyLevel. Also, the properties created in the code in lines-140-142, as mentioned by Jay, doesn't seem to ever get referenced.

I will look a little deeper into this and then, eventually, merge it back to GitHub. If you just want the changes to get usage data working, here is the revised code for processUpdate (note the old code is just commented out for now):
Code: Select all
   ########################################
   def processUpdate(self, updateDict):
      self.debugLog("processUpdate called")
      updateType = updateDict["updateType"]
      if updateType == "updateDevice":
         #update the states
         deviceInfo = updateDict["device"]
         devAddress = deviceInfo.get("id", -1)
         devId = self.deviceDict.get(str(devAddress), 0)
         dev = indigo.devices.get(devId, None)
         keyValueList = []
         if dev and dev.enabled:
            self.debugLog("processUpdate: found device (%s) updating: %s" % (dev.name, deviceInfo))
            # This first set of if/elif will take care of dimmers, locks, and relays.
            if "level" in deviceInfo:
               dev.updateStateOnServer("brightnessLevel", value=deviceInfo["level"])
               # keyValueList.append({'key':'brightnessLevel', 'value':int(deviceInfo["level"])})
            elif "locked" in deviceInfo:
               keyValueList.append({'key':'onOffState', 'value':bool(int(deviceInfo["locked"]))})
            elif "status" in deviceInfo and dev.deviceTypeId != "veraThermostat":  #some versions of the API send an erroneous status for thermostats which have no on/off state
               keyValueList.append({'key':'onOffState', 'value':bool(int(deviceInfo["status"]))})
            

            # Next, we deal with thermostat and other values
            if u'mode' in deviceInfo:
               keyValueList.append({'key':'hvacOperationMode', 'value':kThermostatModeLookup[deviceInfo["mode"]]})
            if "heatsp" in deviceInfo:
               keyValueList.append({'key':'setpointHeat', 'value':int(deviceInfo["heatsp"])})
            if "coolsp" in deviceInfo:
               keyValueList.append({'key':'setpointCool', 'value':int(deviceInfo["coolsp"])})
            if "temperature" in deviceInfo:
               keyValueList.append({'key':'temperatureInput1', 'value':int(deviceInfo["temperature"])})
            if "fanmode" in deviceInfo:
               keyValueList.append({'key':'hvacFanMode', 'value':kThermostatFanLookup[deviceInfo["fanmode"]]})
            if "batterylevel" in deviceInfo:
               uiString = "%s%%" % deviceInfo["batterylevel"]
               keyValueList.append({'key':'batteryLevel', 'value':int(deviceInfo["batterylevel"]), 'uiValue':uiString})
            if "watts" in deviceInfo:
                dev.updateStateOnServer("curEnergyLevel", value=deviceInfo["watts"])
               # keyValueList.append({'key':'energyCurLevel', 'value':int(deviceInfo["watts"])})
            if "kwh" in deviceInfo:
               dev.updateStateOnServer("accumEnergyTotal", value=deviceInfo["kwh"])
               #keyValueList.append({'key':'accumEnergyTotal', 'value':int(deviceInfo["kwh"])})
            if len(keyValueList) > 0:
               dev.updateStatesOnServer(keyValueList)
            if "state" in deviceInfo:
               if deviceInfo["state"] in veralib.kErrorStates:
                  dev.setErrorStateOnServer("device error")
         else:
            if dev:
               self.debugLog("processUpdate: device with Vera ID %i found (%s) but is disabled, skipping update" % (devAddress, dev.name))
            else:
               self.debugLog("processUpdate: no device with Vera ID %i found, skipping update" % devAddress)
      elif updateType == "deleteDevice":
         self.debugLog("\n\nDELETING DEVICE\n\n")
         # the device disappeared from the vera so we'll want to deal with it
         devAddress = updateDict.get("device", -1)
         self.debugLog("deleting device id: %s" % str(devAddress))
         devId = self.deviceDict.get(str(devAddress), 0)
         dev = indigo.devices.get(devId, None)
         if dev:
            dev.setErrorStateOnServer("device deleted")
            self.errorLog('Device "%s" (id: %s) deleted on the Vera' % (dev.name, devAddress))

Posted on
Wed Aug 30, 2017 6:27 am
MartinG offline
Posts: 116
Joined: Aug 19, 2016

Re: Enabling usage tracking

:D

Time for some hero worship, methinks. Thank-you, sir! (And timeto get a vera-edge on order too)

Looks to me that the easiest way to make these updates would be to install the plugin as normal, then just edit plugin.py directly, making your changes.

Posted on
Wed Aug 30, 2017 6:46 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Enabling usage tracking

Thanks for looking into this.

In regards to keyValueList versus updateStatesOnServer, keyValueList is just a batch of all the states that need to be updated. They are then updated in a single atomic call (dev.updateStatesOnServer(keyValueList)) which is more efficient than updating the states individual especially for the SQL Logger plugin. I think the reason it wasn't working in this case was the state names were incorrect (curEnergyLevel is correct, energyCurLevel is not). That doesn't explain why kWh wasn't working, but maybe it was? You can also pass in a uiValue for that state that includes the units postfix (to be shown on Control Pages). Since we know the units are in W and kWh this would be my recommended (but untested) changes:

Code: Select all
            if "watts" in deviceInfo:
               uiString = "%d W" % int(deviceInfo["watts"])
               keyValueList.append({'key':'curEnergyLevel', 'value':int(deviceInfo["watts"]), 'uiValue':uiString})
            if "kwh" in deviceInfo:
               uiString = "%d kWh" % int(deviceInfo["kwh"])
               keyValueList.append({'key':'accumEnergyTotal', 'value':int(deviceInfo["kwh"]), 'uiValue':uiString})

Also, I'm not sure if kWh is really just an integer (versus float/real) from Vera or not. If it is a real number then for the last 2 lines I'd try:

Code: Select all
               uiString = "%0.2f kWh" % float(deviceInfo["kwh"])
               keyValueList.append({'key':'accumEnergyTotal', 'value':float(deviceInfo["kwh"]), 'uiValue':uiString})

Image

Posted on
Wed Aug 30, 2017 9:09 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: Enabling usage tracking

matt (support) wrote:
...In regards to keyValueList versus updateStatesOnServer, keyValueList is just a batch of all the states that need to be updated. They are then updated in a single atomic call (dev.updateStatesOnServer(keyValueList)) which is more efficient than updating the states individual especially for the SQL Logger plugin. I think the reason it wasn't working in this case was the state names were incorrect (curEnergyLevel is correct, energyCurLevel is not). That doesn't explain why kWh wasn't working, but maybe it was? You can also pass in a uiValue for that state that includes the units postfix (to be shown on Control Pages). Since we know the units are in W and kWh this would be my recommended (but untested) changes:...

Close, very close. But... The real villain was trying to cast the incoming values from the Vera as int. They are strings. I fixed the three I was working on. But, I suspect other similar attempts to cast values as int (thermostat and batteryLevel come to mind) will also fail to update. However, since I don't have a Z-Wave T-stat, I am not going to mess with that. BTW, in setting the UI value I also had to treat the data as a string (%s). %d blew up.

So, for those following along... For example, I just ended up changing
    uiString = "%d W" % int(deviceInfo["watts"])
    keyValueList.append({'key':'curEnergyLevel', 'value':int(deviceInfo["watts"]), 'uiValue':uiString})
    to
    uiString = ("%s W" % deviceInfo["watts"])
    keyValueList.append({'key':'curEnergyLevel', 'value':deviceInfo["watts"], 'uiValue':uiString})
Which brings me to the last issue... this whole processUpdate should probably be wrapped in a Try block with some decent error reporting. Not necessary for the plugin to run, but quite useful to get it running. Maybe I'll get to that.

Now, as to the properties that are defined and never referenced (like, around line 126)... I am not really familiar with factory devices. Could those properties be used in there factory handler to set the appropriate states? If so, where is that done.

Posted on
Wed Aug 30, 2017 10:15 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Enabling usage tracking

berkinet wrote:
Which brings me to the last issue... this whole processUpdate should probably be wrapped in a Try block with some decent error reporting. Not necessary for the plugin to run, but quite useful to get it running. Maybe I'll get to that.


Probably, particularly if people are going to use this as a permanent solution.

berkinet wrote:
Now, as to the properties that are defined and never referenced (like, around line 126)... I am not really familiar with factory devices. Could those properties be used in there factory handler to set the appropriate states? If so, where is that done.


So, those sections are what controls whether a device does or does not support reporting energy. Uncomment them, then when the device is created it looks to see if those properties were retrieved from the Vera and if so it adds the associated properties. That's why just changing the code doesn't effect the devices you already created - it only happens when the device is created. I'm not sure that the logic in there is totally correct, but i suspect that uncommenting them then creating a new device will make it work.

Fork the github repo, apply your patches, and issue a pull request and I'll accept and create a new release for it (don't forget to bump the version number).

I wonder why casting wasn't working if they were just numbers in the strings...

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Wed Aug 30, 2017 10:22 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: Enabling usage tracking

jay (support) wrote:
...I wonder why casting wasn't working if they were just numbers in the strings...

Me too. I guess I could have tried float, but since doing nothing (I.e., a string) worked, I figured, why mess with it.

Posted on
Wed Aug 30, 2017 10:25 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Enabling usage tracking

Note the edit to my post above to answer your questions... ;)

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Wed Aug 30, 2017 10:27 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Enabling usage tracking

berkinet wrote:
jay (support) wrote:
...I wonder why casting wasn't working if they were just numbers in the strings...

Me too. I guess I could have tried float, but since doing nothing (I.e., a string) worked, I figured, why mess with it.


I wonder if that's going to cause problems in other areas - like when you try to compare a state value. It may be that those values are going to be treated as strings in comparisons rather than as numbers. You could put a try block just around the cast (put it into a variable rather than in-line) and print the exception to see why it's failing...

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Aug 31, 2017 2:43 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: Enabling usage tracking

jay (support) wrote:
...I wonder if that's going to cause problems in other areas - like when you try to compare a state value. It may be that those values are going to be treated as strings in comparisons rather than as numbers. You could put a try block just around the cast (put it into a variable rather than in-line) and print the exception to see why it's failing...

Ok, some progress, and new questions...

After adding the following exception handler:
Code: Select all
except Exception, e:
               self.logger.exception(u"Error encountered in processUpdate")
               return
and setting the uiString definition back to using %d and int, I get
    Vera Bridge Error Error encountered in processUpdate
    Traceback (most recent call last):
    File "plugin.py", line 321, in processUpdate
    uiString = ("%d W" % int(deviceInfo["watts"]))
    ValueError: invalid literal for int() with base 10: '0.000'
However, changing to a float worked, and I'll leave it that way for now:
Code: Select all
uiString = ("%2.2f W" % float(deviceInfo["watts"]))
But, I had one interesting observation. If I changed the name of a state to something nonexistent, the error was reported, but the handler was not invoked, just a system error was reported, twice. The update code is:
Code: Select all
   if len(keyValueList) > 0:
                  dev.updateStatesOnServer(keyValueList)
and the logged message was:
    Error device "Pool Patio Overhead" state key cuEnergyLevel not defined (ignoring update request)
    Error device "Pool Patio Overhead" state key cuEnergyLevel not defined (ignoring update request)
Note that without the handler, no error is logged at all, so, something is being trapped. Just curious as to why the exception handler was not executed?

Posted on
Thu Aug 31, 2017 9:02 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Enabling usage tracking

berkinet wrote:
But, I had one interesting observation. If I changed the name of a state to something nonexistent, the error was reported, but the handler was not invoked, just a system error was reported, twice. The update code is:
Code: Select all
   if len(keyValueList) > 0:
                  dev.updateStatesOnServer(keyValueList)
and the logged message was:
    Error device "Pool Patio Overhead" state key cuEnergyLevel not defined (ignoring update request)
    Error device "Pool Patio Overhead" state key cuEnergyLevel not defined (ignoring update request)
Note that without the handler, no error is logged at all, so, something is being trapped. Just curious as to why the exception handler was not executed?


The error you see when using a state key that doesn't exist comes from the server, not from the plugin (so your error handler isn't called). So when your code does the dev.updateStatesOnServer() message, the update list is sent to the server. It iterates through it and when it sees a state key that isn't currently there, it throws that error (note that it's reported as a general error rather than a plugin specific error)..

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Aug 31, 2017 9:11 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: Enabling usage tracking

jay (support) wrote:
...The error you see when using a state key that doesn't exist comes from the server, not from the plugin (so your error handler isn't called). So when your code does the dev.updateStatesOnServer() message, the update list is sent to the server. It iterates through it and when it sees a state key that isn't currently there, it throws that error (note that it's reported as a general error rather than a plugin specific error)..

Obvious when I think about it... which I didn't. :oops: . Thanks. I assume there is no way for the server to know which plugin requested the state update, and report that information (otherwise you'd have done that, right).

Posted on
Thu Aug 31, 2017 9:24 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Enabling usage tracking

berkinet wrote:
I assume there is no way for the server to know which plugin requested the state update, and report that information (otherwise you'd have done that, right).


That error will most often only be thrown when developing a plugin, and even if it's not it's easy enough for a user to identify the device (therefore the plugin) and report it that way.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Aug 31, 2017 9:38 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Enabling usage tracking

Version 2.0.2 released on GitHub with @berkinet's energy usage additions. Thanks for the contribution!

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Aug 31, 2017 10:17 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: Enabling usage tracking

berkinet wrote:
jay (support) wrote:
...The error you see when using a state key that doesn't exist comes from the server, not from the plugin (so your error handler isn't called). So when your code does the dev.updateStatesOnServer() message, the update list is sent to the server. It iterates through it and when it sees a state key that isn't currently there, it throws that error (note that it's reported as a general error rather than a plugin specific error)..
Obvious when I think about it... which I didn't. :oops: ...

BTW, I figured out the reason the state updates with the misspelled state did not throw errors before . It was simply because the code never got to that point. It blew up on the attempt to cast the incoming data as an int and silently returned without ever adding the data to keyValueList and never called updateStatesOnServer.

Posted on
Fri Sep 01, 2017 3:04 am
MartinG offline
Posts: 116
Joined: Aug 19, 2016

Re: Enabling usage tracking

jay (support) wrote:
Version 2.0.2 released on GitHub with @berkinet's energy usage additions. Thanks for the contribution!


That's fantastic - thanks to both of you.

Now I just need to get the VeraEdge working - have to say that has been a shockingly poor experience so far.

Who is online

Users browsing this forum: No registered users and 1 guest

cron