New Plugin: GhostXML

I spent a fair bit of time searching available options and flatDict seemed to be the best available option.

The plugin is for the community to develop together, but my own personal preference would be to avoid expanding it to cover what might be one-off solutions (admittedly, a json array is probably not a one-off solution) when flatDict is expecting a json object. It wouldn't be too tough for a user to create a script to snag a copy of the data, convert it to a json object and save it to a local file--which could then be easily read by the plugin.

I had a far briefer look I'm sure and couldn't find anything else either.

My current thoughts though are why go to Json --> Dict --> Flat Dict; isn't there an option to go straight from Json --> Flat Dict ?
(bearing in mind that I struggle a bit with python (particularly the editing environment )- mainly preferring C#)

Glenn

DaveL,

Well took a while; but hopefully found a very lightweight suitable solution.

I found that if we checked to see whether Json array and then could flatten once down to dict before as normal running FlatDict to get desired result.

Then I ran into issue with IndigoDeviceNames - presumably not being allowed to start with a number. Managed to overcome that by adding No_ to the front of the key - although that took a bit of time.

Code: Select all

            if isinstance(parsed_simplejson, list):
                self.debugLog("List Detected - Flattening to Dict")
                parsed_simplejson = dict(("No_"+str(i),v) for  (i,v) in enumerate(parsed_simplejson))


With this addition in parseTheJSON - checks for list from json - and if so - flattens once before then running flattendict - which flattens again appropriately on my testing.(essentially using the same code - which is where I got it from)

Sorry - still sorting out Github - but will upload soon to have a look at : Here now, and have issued pull request if looks good.
https://github.com/Ghawken/GhostXML




Digest Auth working fine with these changes.

Changes:
- add Digest Authentication options for each device (devices will need to be resaved on upgrade)
- username/password options in each device options
- add check for Json list and appropriately enables
- checks for false and true and changes to Python standard False & True


Glenn

Wow Glenn - you were busy! Thank you very much for the contribution. I will take a look at your changes as soon as I can.

The issue with keys that start with a number is a great catch that should've been in the plugin from the outset. I may make a suggestion to move that bit to the cleanTheKeys() method as it's something that could be universal. I agree that converting the list belongs in parseTheJSON() as it shouldn't ever be an issue with XML.

I'm sure that the digest authentication bit will be hugely helpful and is a great addition.

I received a pull request from Git, so it looks like you've submitted it successfully. Cheers!
Dave

Hi Glenn - I just sent you a PM and a note through Git.

Please let me know what you think.
Dave

Hi,

Thanks for this great plugin! just starting to use it with a Nuki doorlock and it works great.

One thing I noticed in the Indigodomo GUI is when I set a device to auto refresh (15 sec....) the device get the status disabled for a (very) short time.
Nothing serious only cosmetic I think.

Regards,

Wouter

Thanks Wouter. I'll add a note to take a look at this next time I'm in the code.

Glad you like the plugin!
Dave

Are you getting this response by entering the URL into a browser, or are you using a script to get the response?
Is this all the information that's ever returned?

I'm making a few assumptions here (for example, we assume that the return is always separated by commas, that the key/value pairs are always separated by an equals sign, and that neither a ',' nor a '=' will appear elsewhere in the response), but this simple script gets you 90 percent of the way there:

Code: Select all

response = 'ret=OK,htemp=18.0,hhum=-,otemp=10.0,err=0,cmpfreq=0'
print(response)

# Split the result into bits
split_response = response.split(',')
print(split_response)

# Split the bits into variable/value pairs
new_list = []
for element in split_response:
    new_list.append(element.split('='))
print(new_list)

# Convert the variable/value pairs into something json-y
new_dict = {}
for element in new_list:
    new_dict[element[0]] = element[1]
print(new_dict)

Result:

Code: Select all

ret=OK,htemp=18.0,hhum=-,otemp=10.0,err=0,cmpfreq=0
['ret=OK', 'htemp=18.0', 'hhum=-', 'otemp=10.0', 'err=0', 'cmpfreq=0']
[['ret', 'OK'], ['htemp', '18.0'], ['hhum', '-'], ['otemp', '10.0'], ['err', '0'], ['cmpfreq', '0']]
{'otemp': '10.0', 'hhum': '-', 'err': '0', 'cmpfreq': '0', 'ret': 'OK', 'htemp': '18.0'}


Note that this code is set up to run in a python shell, and not yet ready to run in Indigo. Getting the HTML response and parsing the data to variables would be trivial once we are sure we know what the response will look like.

Thanks much. Got it working. HTML Response in Indigo Variables. Not as shell but in Indigo.

viewtopic.php?f=107&t=17141&p=131479#p131479

One thing I noticed in the Indigodomo GUI is when I set a device to auto refresh (15 sec....) the device get the status disabled for a (very) short time.

Thanks Wouter. I'll add a note to take a look at this next time I'm in the code.

Dave,

Just had a look at this; the only time the UI is consciously set to "Disabled" is in deviceStopComm()

I suspect that dev.StateListOrDisplayStateIdChanged() is internally calling (or causing) deviceStopComm() and deviceStartComm() - and that's called from fixErrorState() which is one of your bits so I'll leave you to dig further on that

Peter

Dave,

Just had a look at this; the only time the UI is consciously set to "Disabled" is in deviceStopComm()

I suspect that dev.StateListOrDisplayStateIdChanged() is internally calling (or causing) deviceStopComm() and deviceStartComm() - and that's called from fixErrorState() which is one of your bits so I'll leave you to dig further on that

Peter

Thanks Peter. If stopComm() is the only place that's done, I think the solution is to change "disabled" to something more apropos like "comms stopped" and writing it at debug level instead of at log level. Would you agree?

Thanks Peter. If stopComm() is the only place that's done, I think the solution is to change "disabled" to something more apropos like "comms stopped" and writing it at debug level instead of at log level. Would you agree?

Don't forget stopComm() is also called if someone DOES disable comms in the UI - whatever we change the state to ("Disabled", "Comms stopped" or otherwise) it won't answer Wouter's point that the device is for some reason stopping/starting on a regular basis.

(If we don't set the state to something, then the state/UI still shows "Enabled" even when it isn't; if we set it to "Comms stopped" then Wouter will still notice the temporary stop/start)

I've just realised whats actually happening - it's not in fixErrorState() as I thought earlier.

Because our state list is always varying, dependent on the keys returned by the XML/JSON, we always call dev.stateListChangedOrUpdated() when we have a finalDict. Thus EVERY time we refresh data, we indirectly stop and start our devices.

We should code this better to check (roughly):

Code: Select all

if (stateKeysBeforeRefresh == stateKeysAfterRefresh):
 updateStateOnServer(...every state value...)
else:
 dev.stateListChangedOrUpdated()

This will only stop/start the device if the keys change, which they shouldn't do much after the first successful download. Other times it will just update the states. I know there's a new API for newStates = [this:that,a:b,abc:xyz] followed by updateAllTheseStatesOnTheServer(newStates) but I've not used it yet.

Peter

Hi Peter - I've had a chance to think about this a bit (real life has been interfering lately). Anyway, I have added a simple logic test in the getDeviceStateList() method. However, the rub is that this method is actually called by stateListOrDisplayStateIdChanged() automatically, so the test (and ultimate fix) will need to be moved elsewhere.

I also added a note to that iTimer variable we chatted about earlier--I actually think that reference may simply be deleted. In fact, I think the entire try/except block can be deleted (save the variable assignment in the try)--I didn't want to do that out of hand, however, since it was your addition.

Cheers,
Dave

Hi,

I'm sort of close to having what I need I think but I'm not able to get the list of device states. Here is what I'm getting in my debug log:

Code: Select all

   GhostXML Debug                  Device needs updating set to: True
   GhostXML Debug                  getDeviceStateList() method called.
   GhostXML Debug                  Pulling down existing state list.
   GhostXML Debug                    Writing dynamic states to device.
   GhostXML Debug                  [u'pumps_pump8', u'pumps_pump5', u'pumps_pump4', u'pumps_pump7', u'pumps_pump6', u'pumps_pump1', u'pumps_pump3', u'pumps_pump2']
   GhostXML Debug                  [u'pumps_pump8', u'pumps_pump5', u'pumps_pump4', u'pumps_pump7', u'pumps_pump6', u'pumps_pump1', u'pumps_pump3', u'pumps_pump2']
   GhostXML Debug                  True
   GhostXML Debug                  Device needs updating set to: False
   GhostXML Debug                  parseStateValues() method called.
   GhostXML Debug                  Writing device states:
   GhostXML Debug                     pumps_pump1 = 0,0,8
   GhostXML Debug                     pumps_pump2 = 0,0,0
   GhostXML Debug                     pumps_pump3 = 0,0,32768
   GhostXML Debug                     pumps_pump4 = None
   GhostXML Debug                     pumps_pump5 = None
   GhostXML Debug                     pumps_pump6 = None
   GhostXML Debug                     pumps_pump7 = None
   GhostXML Debug                     pumps_pump8 = None
   GhostXML Debug                  PumpsXML updated.

At this point however I'm not getting presented with a list of variables. I'm not sure how to pull the variable state out of this. Do you see something wrong with my log above?

Thanks in advance.

At this point however I'm not getting presented with a list of variables. I'm not sure how to pull the variable state out of this. Do you see something wrong with my log above?

Thanks in advance.

Hi Tornado - based on the log dump above, it seems like the plugin is working properly.

I want to make sure I understand what you mean when you say that you're not getting a list of variables and that you can't pull the variable state out of this. The plugin doesn't present any variables, but rather saves the values as custom device states. When you look at the custom device state panel, what do you see? (It's located below the Device Details panel.)

Screen Shot 2017-02-23 at 6.41.29 AM.png (48.23 KiB) Viewed 3808 times

When you try to add the device to a control page, what do you see?

Screen Shot 2017-02-23 at 6.43.48 AM.png (51.49 KiB) Viewed 3808 times

I want to make sure I understand what you mean when you say that you're not getting a list of variables and that you can't pull the variable state out of this. The plugin doesn't present any variables, but rather saves the values as custom device states. When you look at the custom device state panel, what do you see? (It's located below the Device Details panel.)

Embarrassed to say I never knew that was there. Now of course I can see everything. Thanks a bunch for the plugin and the assistance!

Don't feel bad. Ever since Apple did away with scroll bars, lots of people miss that panel. Glad to hear all is well.

Cheers.


Sent from my iPhone using Tapatalk