New Plugin: GhostXML

This is going to take a bit more digging. Based on my initial digging:

1. I can establish GhostXML devices based on both sets of XML and bot devices update without error. (GhostXML v0.2.01)
2. Interestingly, the plugin is not parsing 100 percent of the Sitesage channel nodes (you'll notice for example that channel "0" and all the "A" - "C" channels are not being returned at all.)

I'll need to dig deeper.

Dave

In the "error" case, the A, B, and C channels are being returned interspersed with the numeric ones; for example:

GhostXML Debug channels_channel_16_CT_size = error
GhostXML Debug channels_channel_16_Number = A1
GhostXML Debug channels_channel_16_label = error
GhostXML Debug channels_channel_16_real_power = error
GhostXML Debug channels_channel_16_va = error

There are always 45 possible channels (of which my gateway has 24 set up) in the Sitesage system, and I have no idea what 0 is supposed to represent anyway; there's no way to provide an input to it.

From the way it is acting, it is like when the ?podID=0 gets added to the URL in Ghost, the URL is not being properly posted to the gateway... and the gateway is very particular about the format... when I accidentally sent a ?PodID=0 typing it in manually to Chrome it hung up just like it did originally when the packets were being blocked.

Let me know if there is anything I can try as far as altering the form of the URL, although it will have to wait till I get home to play with it; there's currently no way for me to get any external access through the firewall.

I finally got a chance to look at this a bit more, and of course, you're right. The plugin seems to be picking up all 45 channels (including channel 0 whatever that is) so that's a good sign.

So then the question becomes one of why is the plugin bombing at all? I can't reproduce your error exactly, but I have been able to produce a similar error. It could be the process I'm using to download the data (Curl); I'll need to do some more testing.

Thanks for your patience; I think we're zeroing in on a solution.
Dave

I was looking at your iterateXML.py source earlier and (while I am not as up on python source as I am AppldScript and C++), I'm wondering if you need another elif in the

Code: Select all

            if len(element):
                aDict = XmlDictConfig(element)
               
                if element.items():
                    aDict.updateShim(dict(element.items()))
                self.updateShim({element.tag: aDict})
               
            elif element.items():
                self.updateShim({element.tag: element.text}) # This line added to handle when value and attribs are both present.
                elementTagAttribs = element.tag + u'_A_t_t_r_i_b_s' # This line added to create a unique element.tag for attribs.
                self.updateShim({elementTagAttribs: dict(element.items())}) # This line modded to use new element.tag + '_Attribs'.

            else:
                self.updateShim({element.tag: element.text}) # WAS: _self.updateShim({element.tag: element.text.strip()})_ with strip(), the function will choke on some XML. 'NoneType' object has no attribute 'strip'.


area something like

Code: Select all

            if len(element):
                aDict = XmlDictConfig(element)
               
                if element.items():
                    aDict.updateShim(dict(element.items()))
                self.updateShim({element.tag: aDict})
               
            elif element.items():
                self.updateShim({element.tag: element.text}) # This line added to handle when value and attribs are both present.
                elementTagAttribs = element.tag + u'_A_t_t_r_i_b_s' # This line added to create a unique element.tag for attribs.
                self.updateShim({elementTagAttribs: dict(element.items())}) # This line modded to use new element.tag + '_Attribs'.

            elif len(element.text):
                self.updateShim({element.tag: element.text}) # WAS: _self.updateShim({element.tag: element.text.strip()})_ with strip(), the function will choke on some XML. 'NoneType' object has no attribute 'strip'.

            else:
                self.updateShim({element.tag: u'_NULL'}) # To avoid possible throw on aDict having no attributes or type in updateShim...



Interesting. Since I can't reproduce your exact error, would you mind giving that a go and see what happens? I did a ton of research on ways to parse the data and none were perfect. Your suggested revision looks to be a good enhancement. If it works for you, we can get it up on GitHub.

Dave

Well, adding that code generated a syntax error at the u'_NULL' line, so I guess I'll have to dig a little deeper into what python wants for that second updateShim parameter... but I agree that it's got to be the empty value field that's breaking the iteration and hitting the exception clause.

Please let me know if you find anything that works. Thanks!

Dave


Sent from my iPhone using Tapatalk

Please let me know if you find anything that works. Thanks!

Dave


Sent from my iPhone using Tapatalk

FOUND IT!!!

In line 358 of plugin.py you test for whether it is an XML file by looking whether the URL ends with 'xml'... which the ?podID=0 defeats. When I set the else clause to assume XML, everything works, but if you know the python function to search for containment anywhere within the URL (or better yet in the first line of the data stream), that's obviously a better fix.

Excellent. Good find.

That was pretty short sighted of me to assume that the URL would always end with 'xml' or 'json'. I agree that looking at whether the URL contains 'xml' is too dangerous; this would cause problems: http://www.foo.com/xml.json

Looking at the first line of the return seems like a good approach to identifying XML--it's hard to imagine that a valid json return could contain '<?xml' at the start of its first line. Another alternative would be to ask the user to specify what type of return to expect in a device config dropdown. It would allow us to be certain, but also allow for future expansion to other types of returns (what ever those might be.) What do you think?

Thanks for the detective work on this. I'll try to get an update posted to the repo this weekend. Cheers!
Dave

Given an actual address (excluding parameters) should always end in .xml, you should be able to do the python equivalent of:

substring($url,0,search($url,"?")-1)

That's the PHP to get a URL string up to but excluding the first (optional) question mark. This string should then end in .xml

Though are you handling (or intending handling) XML streams returned by say a PHP script?

http://www.xyz.com?getXML.php?id=1


Sent from my iPhone using Tapatalk

I think we'd be interested in handling XML and JSON returns no matter how they're served. I think I'm going to include a device configuration dropdown box to specify whether the source URL is XML or JSON. Then, it hopefully doesn't matter how we get it--the only important thing being how it's returned.

Dave

While you're in there, could you consider making the polling interval device-specific instead of at the plugin level?

While you're in there, could you consider making the polling interval device-specific instead of at the plugin level?

Hi - that is definitely on the wish list. It's a bit non-trivial, but not insurmountable.

Just an FYI to everybody - I released this plugin to the community via GitHub (differently than the other plugins I maintain) so that anyone can contribute. (Not suggesting that you should code it yourself! ) But if there is someone with the chops and the time to handle independent device refresh rates, feel free to let me know and we'll set you up to have a crack at it.

Dave

I've updated the plugin to use explicit selection of whether the feed is JSON or XML. There is no error testing to see if the user has entered the settings properly; if the user enters a JSON feed URL and selects a Feed Type of XML, the plugin will likely bomb.

This is a short-term fix to lift the requirement that all URLs end with 'XML' or 'JSON' which will allow users to use a broader array of URLs.

Please let me know if it doesn't behave the way you expect it to.
Dave

You can download it here: GhostXML Plugin.

Small update to correct fatal error in data typing.

You can download it here: GhostXML Plugin.