Extending "valuesDict" in callback functions

I read your first post multiple times but I'm still not getting the picture of what you're trying to do overall (vs how you'd like to implement it).

Sure. The plugin I'm writing will simply plot some data, such as temperatures from sensor devices. As an example, the end result would be a Control Page with a plot showing temperatures over time for each of the rooms in my house. Each 'device' would contain all of the data needed to create the plot.

See the attachment for an early example of what the plot might look like.

So when configuring the device, the user would enter information such as Plot Title, Start Date/Time, End Date/Time, etc. The user would also create a list of devices and state keys to plot, where each device would represent a line graph or bar graph in the plot.

The challenge I'm running into is creating a good UI for the user to create the LIST of devices and state keys to graph. So for each item in the list, the user would select the device, select the state key to graph and format options such as line color, line width, etc. A single plot (device) will likely contain multiple plots. I'm trying to mimic the UI in HomeKit Bridge for when the user is adding devices and actions that will be shared by HomeKit.

I've also uploaded a screenshot of what the UI looks like at this point. (Note that the checkbox with no label is usually hidden.)

Doing it in deviceUpdated() is going to break badly because that gets called for EVERY change to EVERY Indigo device. Not just your plugin's devices. Your code doesn't even check to see if the device belongs to your plugin.

Slight correction: it will get called for all devices if you subscribe to device changes. Otherwise you just get changes to devices owned by your plugin.

I did not know that. I didn't realize it would do anything without calling the subscribe function.

[jay (support)]

I did not know that. I didn't realize it would do anything without calling the subscribe function.

Makes more sense now, right? Point of fact, that's the only way it worked before we added the ability to subscribe to other changes.

[jay (support)]

I've also uploaded a screenshot of what the UI looks like at this point. (Note that the checkbox with no label is usually hidden.)

A few clarifying questions:

1. What does the Add Plot Information button do?
2. What does the Save Plot button do?
3. When you use the word plot, what exactly do you mean? A single state showing its behavior over time?
4. When you have multiple plots, does that mean you're adding multiple #3 to the same graph?
5. Do you anticipate having the same state represented in a different way in a different graph?

Using different terminology, you want to create a graph which has multiple series (state values from potentially from different devices) over the period of time specified shown as lines on the graph. Do I have it right?

I've also uploaded a screenshot of what the UI looks like at this point. (Note that the checkbox with no label is usually hidden.)

A few clarifying questions:

What does the Add Plot Information button do?

The user selects an action from the Action popup menu. Options are: Add New Plot, Edit Plot and Delete Plot. (Note that "Plot" really should be "Series") Depending on what action they select, the button below changes. In the case of the example I posted, the user selected "Add New Plot" so the button shown (not hidden) is "Add Plot Information". When the user clicks the button, the bottom half of the dialog is enabled and default values are provided.

What does the Save Plot button do?

The Save Plot button confirms the data just entered in the Plot Data Settings area of the dialog.

When you use the word plot, what exactly do you mean? A single state showing its behavior over time?

I've been struggling with terminology, but you've cleared that up for me below. In this case, "plot" really should be "series".

When you have multiple plots, does that mean you're adding multiple #3 to the same graph?

As you stated below, multiple plots are actually multiple series. So the device represents the graph and within the graph is a list of series - the list of dicts I've been talking about in this thread.

Do you anticipate having the same state represented in a different way in a different graph?

Hmmmm... not quite sure where you are going with this. Originally, the state was going to be either Temperature or Humidity from a sensor. The idea being to compare temperatures or humidity values from different sensors over time. However, as I've gotten further into the project, I'm realizing that there are other states that would make sense to graph, such as battery level. (Still thinking on this one)

Using different terminology, you want to create a graph which has multiple series (state values from potentially from different devices) over the period of time specified shown as lines on the graph. Do I have it right?

Exactly! As I mentioned above, I've been struggling with the correct terminology, but you've summed it up quite well. Thanks! I'll definitely make changes to the UI along those lines..

Creating the graph with MatPlotLib is actually turning out the be the easy part. The hard part is creating the interface. As I mentioned in a previous post, what I'd like is the ability to create a more traditional UI for editing a list of items. So for the example we've been discussing, the top half (General Plot Settings) and the bottom half (Plot Data Settings) would be separate dialogs. In place of the Action menu and button below it, there would be three buttons side by side: Add, Edit and Delete. Edit and Delete would only be enabled if something in the list was selected.

If the user clicked the Add button, a new dialog would pop open allowing the user to enter information about the series. And at the bottom of that dialog would be two buttons side by side: Save and Cancel.

If the user clicked on an item (series) in the list, the Edit and Delete buttons would become enabled. If the user clicked Edit, the same dialog as before would pop up allowing the user to change settings for the selected series. They would then click the Save button or the Cancel button.

There are other ways to implement editing of a list, but I've found that HomeKit Bridge is the most intuitive for me. I'm trying to mimic that UI, but even more intuitive.

-Tommy

You can't do that in validateDeviceConfigUi or closedDeviceConfigUi because at the point those are called, the device object may not exist yet, especially for devices that are currently being created.

Ok, that makes sense. Although, I'm pretty sure this still didn't work even after I created the device and closed the settings dialog, then went back later to edit the device settings.

You should be doing this in deviceStartComm().

Unfortunately, deviceStartComm() doesn't get called at the appropriate times for my purpose. When the user finishes editing the device settings, when the dialog is closed (and not canceled), I need to convert the json string in valuesDict to a device property in pluginProps. When creating a device for the first time, deviceStartComm() gets called before the configuration is complete. Also, the user may come back and edit the settings at a later date. I'm not completely sure, but I don't think deviceStartComm() will fire at the appropriate time in that scenario either.

Doing it in deviceUpdated() is going to break badly because that gets called for EVERY change to EVERY Indigo device. Not just your plugin's devices. Your code doesn't even check to see if the device belongs to your plugin.

As Jay said, it doesn't get called for all devices if I don't subscribe to device changes. I've actually got this working now (really!) using deviceUpdated(). There is an important if statement checking if the json string exists and is not empty. Then after it saves the property in pluginprops, it sets the json string to an empty string. This is how I know during a device update whether or not I need to resave the property again.

This doesn't seem like the most straight forward way of handling this, but it works... and I can't think of another way to handle it.

Also, the user may come back and edit the settings at a later date. I'm not completely sure, but I don't think deviceStartComm() will fire at the appropriate time in that scenario either.

deviceStartComm() will get called every time that the configuration dialog is saved (deviceStopComm() will get called first) , unless you specifically tell Indigo not to do so by implementing the didDeviceCommPropertyChange() method.

[jay (support)]

Ok. So, first things first: you can store arbitrarily complex hierarchies of Indigo Lists and Dicts along with python primitives in a device's plugin props (just take a look at a Z-Wave device's ownerProps - they are pretty complex). So I'm not sure why you couldn't get that working (if in fact that was the issue). It may be that the Z-Wave plugin builds those in a different way, but as Matt's the Z-Wave dude I can't really answer that. He might be able to jump in here in the next few days.

I would recommend using a UI that's similar to the Global Property Manager's Manage Global Device Properties menu item. It's not for a plugin specified device, but if you think about each device selected in the top popup as representing the graph device, then you can envision the rest of the dialog managing each series but in the device edit dialog for your graph device type. Feel free do dig through that plugin's source, though it's really quite small/simple. That might be a good starting framework for what you're looking to do. It stores the necessary stuff as properties on the device itself and not as part of a device that's created by the plugin, but I think the concepts might help you along a bit.

Ok. So, first things first: you can store arbitrarily complex hierarchies of Indigo Lists and Dicts along with python primitives in a device's plugin props (just take a look at a Z-Wave device's ownerProps - they are pretty complex). So I'm not sure why you couldn't get that working (if in fact that was the issue). It may be that the Z-Wave plugin builds those in a different way, but as Matt's the Z-Wave dude I can't really answer that. He might be able to jump in here in the next few days.

I can get that working (saving complex hierarchies in plugin props), just not from within validateDeviceConfigUi() or closedDeviceConfigUi(). I couldn't even write a simple string directly to plugin props from either of those methods. It's certainly possible I was doing something wrong, but the exact same code is working from within deviceUpdated().

As a test, I just modified "Example Device - Custom" with only the following code change in validateDeviceConfigUi():

Code: Select all

   def validateDeviceConfigUi(self, valuesDict, typeId, devId):
      # If the typeId is "scene", we want to clear the selections on both
      # dynamic lists so that they're not stored since we really don't
      # care about those.
      self.debugLog(u"validateDeviceConfigUi: typeId: %s  devId: %s" % (typeId, str(devId)))
      if typeId == "scene":
         if "memberDeviceList" in valuesDict:
            valuesDict["memberDeviceList"] = ""
         if "sourceDeviceMenu" in valuesDict:
            valuesDict["sourceDeviceMenu"] = ""

      # Added code here
      dev = indigo.devices[devId]
      localPropsCopy = dev.pluginProps
      localPropsCopy['testProp'] = u"ABCD"
      dev.replacePluginPropsOnServer(localPropsCopy)
      # End of added code

      return (True, valuesDict)

The property 'testProp' does not get saved in the devices pluginProps. It fooled me at first because after dev.replacePluginPropsOnServer, I refreshed the dev from server and checked for the property. It was there! However, from the scripting shell, it does NOT show up if you print the dev.

Code: Select all

>>> dev = indigo.devices[810913725] # "ABC"
>>> print dev


I would recommend using a UI that's similar to the Global Property Manager's Manage Global Device Properties menu item. It's not for a plugin specified device, but if you think about each device selected in the top popup as representing the graph device, then you can envision the rest of the dialog managing each series but in the device edit dialog for your graph device type. Feel free do dig through that plugin's source, though it's really quite small/simple. That might be a good starting framework for what you're looking to do. It stores the necessary stuff as properties on the device itself and not as part of a device that's created by the plugin, but I think the concepts might help you along a bit.

That UI is just another way for working around the fact that you can't locate buttons side-by-side or open up a separate dialog to enter additional information. Personally, I much prefer the UI in HomeKit Bridge over this UI. Believe me, I've looked at the UI (and code) in a lot of plugins to see how lists are implemented. There is a standard way (in computing) of editing a list of: properties, contacts, songs, graph series, or whatever, that you just can't do with the current implementation of Indigo.

A perfect example of the standard way of editing a list is the main Indigo window. There's a "New..." button, an "Edit.." button and a "Delete..." button. There's even a bonus "Duplicate..." button. The buttons are side-by-side. The idea being that if you click "New...", a separate dialog opens up for the user to enter all of the information for the new Device (or Trigger, etc.). The user can either save that device or cancel. If instead the user selects an existing device and clicks "Edit...", the same dialog pops up, and information specific to the existing device can be edited. And of course "Delete..." deletes the item.

As I mentioned earlier in this thread, I would like to request new features that would allow what I'm calling a standard way of editing a list. The first two feature requests would be positioning buttons side-by-side and being able to open another dialog box for editing data, with "Save" and "Cancel" buttons side-by-side.

There are a couple of things that I don't particularly care for in Global Property Manager's UI. First, it states that "Note that all changes made in this dialog are immediately stored and can't be undone." So basically, when you click the "Add Property" button, it is stored. One main goal (which I started this thread with) is to be able to cancel all edits by clicking the Cancel button when closing the Edit Device Settings UI. That's why I am using valuesDict to store the complex data type (as a json string) until I later write it as a property in pluginProps. This happens after the user presses the Save button when closing the Edit Device Settings UI.

The other thing is that once you select a device, you then use the 3rd section of the dialog (bottom section) to add a property. That's not intuitive to me. I would expect to use the 2nd section (middle section). But then the second section is used to edit the list of global properties that are already defined.

I'm not saying this UI is bad in any way, I just prefer the UI in HomeKit Bridge because it most closely resembles the standard way of editing a list of things.

The UI I'm creating is mostly working now. I still have a few more things to do. Since this thread has gone is so many directions, here's a summary of the important things I've learned through the thread:

1. You can't save complex properties to valuesDict (Ex. a list of dicts).
2. If you need to save a complex property to valuesDict, use json to create a string to save.
3. json does not work with indigo.List() or indigo.Dict().
4. You can only update device plugin properties using valuesDict from within validateDeviceConfigUi() or closedDeviceConfigUi(). You cannot update them directly with dev.replacePluginPropsOnServer().

And finally, deviceStartComm() is not being called when the user edits an existing device. I'm still trying to figure this out because FlyingDiver says it should be called.

[jay (support)]

Code: Select all

   def validateDeviceConfigUi(self, valuesDict, typeId, devId):
      # If the typeId is "scene", we want to clear the selections on both
      # dynamic lists so that they're not stored since we really don't
      # care about those.
      self.debugLog(u"validateDeviceConfigUi: typeId: %s  devId: %s" % (typeId, str(devId)))
      if typeId == "scene":
         if "memberDeviceList" in valuesDict:
            valuesDict["memberDeviceList"] = ""
         if "sourceDeviceMenu" in valuesDict:
            valuesDict["sourceDeviceMenu"] = ""

      # Added code here
      dev = indigo.devices[devId]
      localPropsCopy = dev.pluginProps
      localPropsCopy['testProp'] = u"ABCD"
      dev.replacePluginPropsOnServer(localPropsCopy)
      # End of added code

      return (True, valuesDict)

The property 'testProp' does not get saved in the devices pluginProps. It fooled me at first because after dev.replacePluginPropsOnServer, I refreshed the dev from server and checked for the property. It was there! However, from the scripting shell, it does NOT show up if you print the dev.

That would be because while you were successful in updating the device temporarily (which is dangerous for a variety of reasons), when the dialog is saved, the props are overwritten by the ones returned in the valuesDict, which doesn't contain "testProp".

Did you try just adding it directly to valuesDict? Like:

Code: Select all

   def validateDeviceConfigUi(self, valuesDict, typeId, devId):
      # If the typeId is "scene", we want to clear the selections on both
      # dynamic lists so that they're not stored since we really don't
      # care about those.
      self.debugLog(u"validateDeviceConfigUi: typeId: %s  devId: %s" % (typeId, str(devId)))
      if typeId == "scene":
         if "memberDeviceList" in valuesDict:
            valuesDict["memberDeviceList"] = ""
         if "sourceDeviceMenu" in valuesDict:
            valuesDict["sourceDeviceMenu"] = ""

      # Added code here
      valuesDict['testProp'] = u"ABCD"
      # End of added code

      return (True, valuesDict)

Anything that you add to valuesDict will get added to the device's pluginProps when the changes are saved.

As I mentioned earlier in this thread, I would like to request new features that would allow what I'm calling a standard way of editing a list. The first two feature requests would be positioning buttons side-by-side and being able to open another dialog box for editing data, with "Save" and "Cancel" buttons side-by-side.

Opening cascading dialogs is really complex because of the roundtrip nature of UI dialogs (Client->Server->Plugin->Server->Client), each of those things being totally separate processes. So that's probably not going to happen anytime soon, and probably not until we invent a completely new way to specify dialogs. I suppose multiple buttons in one field might be a nice to have, but stacking them seems like a reasonable workaround.

We understand that there are limitations to UI construction, but remember that it's a balance between ease of use and flexibility. The more we add the more complex building plugin UI would become (and the more complex for us to support). One thing we would really like to do some day is to allow plugins to do their configuration based on web pages, where you'd have a ton more flexibility (this would be in addition to the current mechanism since it's easier than building web pages with javascript and catching HTTP Posts, etc).

The other thing is that once you select a device, you then use the 3rd section of the dialog (bottom section) to add a property. That's not intuitive to me. I would expect to use the 2nd section (middle section). But then the second section is used to edit the list of global properties that are already defined.

Personal preference I suppose, but in your dialog (I assume) that if you select the series in the list at the top it's definition shows up in the bottom - which seems confusing to me given the controls in between. I think limiting controls, and making them clearly visible when they are present, is good dialog design. I think the popup is what really confuses me but perhaps that's because it's just a pict rather than something interactive I can click around on.

Maybe changing the terminology on the dialog will make it less so to me...

[jay (support)]

After a bit more playing, I've discovered that values added to valuesDict don't always get saved. If the value is a simple python object, it gets added. If it's an indigo.List() with simple python values it gets added. If you add any complexity however it seems to fail. Having never tried storing complex hierarchical data like this, the finding is unexpected. I'll discuss with Matt to see if it's a bug or if we intentionally limited it (it's possible there was a reason we decided to do it like this, I just don't recall).

Did you try just adding it directly to valuesDict? Like:

Code: Select all

   def validateDeviceConfigUi(self, valuesDict, typeId, devId):
      # If the typeId is "scene", we want to clear the selections on both
      # dynamic lists so that they're not stored since we really don't
      # care about those.
      self.debugLog(u"validateDeviceConfigUi: typeId: %s  devId: %s" % (typeId, str(devId)))
      if typeId == "scene":
         if "memberDeviceList" in valuesDict:
            valuesDict["memberDeviceList"] = ""
         if "sourceDeviceMenu" in valuesDict:
            valuesDict["sourceDeviceMenu"] = ""

      # Added code here
      valuesDict['testProp'] = u"ABCD"
      # End of added code

      return (True, valuesDict)

Anything that you add to valuesDict will get added to the device's pluginProps when the changes are saved.

Yes, of course, but you can't add a list of dictionaries via valuesDict. The example from "Example Device - Custom" was just to show that you can't directly save a property from within validateDeviceConfigUi() or closedDeviceConfigUi(). Your explanation about pluginProps getting replaced when the function returns valuesDict makes perfect sense. Now I understand why that happens.

Opening cascading dialogs is really complex because of the roundtrip nature of UI dialogs (Client->Server->Plugin->Server->Client), each of those things being totally separate processes. So that's probably not going to happen anytime soon, and probably not until we invent a completely new way to specify dialogs. I suppose multiple buttons in one field might be a nice to have, but stacking them seems like a reasonable workaround.

We understand that there are limitations to UI construction, but remember that it's a balance between ease of use and flexibility. The more we add the more complex building plugin UI would become (and the more complex for us to support). One thing we would really like to do some day is to allow plugins to do their configuration based on web pages, where you'd have a ton more flexibility (this would be in addition to the current mechanism since it's easier than building web pages with javascript and catching HTTP Posts, etc).

I do understand. The reason I'm suggesting the features (or whatever new features would work) is that there are a LOT of plugins that manage a list. This seems to be a very common task and there are a lot of different UIs right now.

Curious... I use visibleBindingId and enabledBindingId to hide/unhide, or enable/disable, the bottom part of the dialog, depending on what the user clicks on. This is how I simulate opening a new dialog. Doesn't that create round trips as well? Would a new dialog be any worse from the round trip sense?

[jay (support)]

Curious... I use visibleBindingId and enabledBindingId to hide/unhide, or enable/disable, the bottom part of the dialog, depending on what the user clicks on. This is how I simulate opening a new dialog. Doesn't that create round trips as well? Would a new dialog be any worse from the round trip sense?

Yes, but they are stateless roundtrips - they go through the round trip without really stopping (pausing for any processing, but that's comparatively short). Opening another dialog would then create a second open-ended round trip (which would then also potentially spawn more round trips), because the first round trip would still be in the middle of a trip. If everything were in the same process, it would be a snap. But since we wanted to sandbox the plugins so a misbehaving plugin would be much less likely to take the whole server down, and because Indigo is a client/server app, it just makes it much more challenging.

Managing lists is one thing, but managing master/detail relationships (using old DB terminology, probably showing my age) is less common...

I always save json.dumps(your dict) as a property/valuesdict string and then load it w json.loads from valuesdict / props

That always works. Is supported and trivial to do. Then you can store any dict/list thing that can be serialized. I have never had anything that did not work. Have to think about something that would not work.


Sent from my iPhone using Tapatalk

[matt (support)]

Having just refreshed my memory on how this works (by glancing at the code ) I can shed a bit more information on the limitations as well explain what is going on under-the-hood.

A device instance's pluginProps (dev.pluginProps) can hold atomic value types (string, integers, bool) and basic Indigo containers (dicts and lists), and those containers can be nested.

valuesDict is a special dict instance that gets passed around (to various callbacks in the plugin) to handle populating and validating UI. In some cases it mirrors the device instance pluginProps, but there are some restrictions. These restrictions filter/strip out of the pluginProps both dicts and complex lists from being added to the valuesDict and passed through to the UI and back out to the plugin:

• All dicts are removed.
• Lists that contain only atomic types (bool, integer, string) are allowed as long as the string doesn't contain a comma or semicolon.

Everything stored in a valuesDict is something that could represent the state of config UI in the settings dialog. There are no UI controls (supported by Indigo) currently that use dicts, so those are filtered out. We do support a UI list and UI popup menu though, so basic lists are not filtered.

Ideally we probably wouldn't have these restrictions – they exist because of the nature of how the data is round tripped from the plugin -> Indigo Server -> Indigo Client -> UI Dialog and then back out. I'll add to our request list to make the round tripping more seamless. In the mean time stick to using basic atomic types and simple lists.

These restrictions apply just to valuesDict used in the UI callbacks. You can use more complex lists and dicts outside of that scope if you are stuffing them directly into the pluginProps (via dev.replacePluginPropsOnServer), but as I believe is noted above that doesn't work from UI callback methods because the device instance may not exist yet (new device being created) and because they will get overwritten when the UI is closed.