New Plugin: GhostXML

Posted on
Fri Jun 08, 2018 3:44 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

Thanks for the extra info. I am going to take a deeper dive on the payload to see if there's anything there. I know that there are sometimes issues with JSON formatting that I've tried to repair/capture, but I don't suspect that's the case here. I don't recall having heard of a situation where the same JSON works from a file but not a online source. The plugin uses the same parsing code for both, so if there's a difference, I suspect it's somewhere in how the curl is processed.

Does your online route use auth?

The traceback you're seeing is from the parseTheJSON() method so the error is firing in that code -- or in the flatdict module it calls.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Jun 09, 2018 6:25 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: New Plugin: GhostXML

DaveL17 wrote:
...Does your online route use auth?....

Yes. However, that was not the immediate problem. The real issue was a self signed SSL certificate. The error was in the log data I sent you, but I missed it. Adding the -k option to the subprocess.Popen call fixed the problem (-vks instead of -vk). However, I am not sure you really want to do that by default. Perhaps allowing insecure connections ought to be a device property.

With that out of the way, I still have problems. But, they are more design and scope issues than errors. The server in question here does not accept basic or digest auth. Instead, you must request a certificate and then include that with each request. Here is some example code they provide
Code: Select all
from urllib2 import Request, urlopen

values = """
  {
    "pwd": "{{PASSWORD}}",
    "remember": 1
  }
"""

headers = {
  'Content-Type': 'application/json'
}
request = Request('https://device.ip:8080/api/4/auth/login', data=values, headers=headers)

response_body = urlopen(request).read()
print response_body
Here is the same thing done with curl in a shell script I wrote
Code: Select all
TOKEN=`curl --include \
   --insecure \
   -s \
   --request POST \
   --header "Content-Type: application/json" \
   --data-binary "{ \"pwd\": \"${PWD}\", \"remember\": 1 }" } https://${ADDR}:8080/api/4/auth/login`
TOKEN2=`echo $TOKEN|cut -d\" -f4`
I am not sure if this method is generalized enough to be of benefit to anyone except a RainMachine owner?

I also have another question, about the calling curl in Popen as opposed to the requests module. I see you we using this a few years ago but didn't't use it in this plugin. I was wondering why. BTW, I may be wrong, but IIRC requests is now included in Indigo.

Posted on
Sat Jun 09, 2018 7:46 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

Yeah, I must admit that I've never fully come up to speed on creating authentication sessions that require a token. There have been other situations where I could've used that feature but never worked it out. I'm in the midst of a refactor to handle the thread safety issue and will add the feature you mention to the TODO list for the plugin after that's in the can.

I used requests before the TLS issue surfaced and went to curl to get around that. IIRC, handling backwards compatibility with requests is non-trivial. I went away from using requests in all my plugins because of this.

I'm not asking you to do this, but if you were thinking of a pull request, you might want to wait until the thread-safe version of the plugin is released. It won't address the issues you're seeing, but will be different enough that a pull off the current version will be tough to integrate.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Jun 09, 2018 9:56 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: New Plugin: GhostXML

DaveL17 wrote:
...I'm not asking you to do this, but if you were thinking of a pull request, you might want to wait until the thread-safe version of the plugin is released....

I might just get inspired (and then again, I might not) to add the insecure option. If I do, I will wait to integrate it until after the next update. I'll also look into supporting auth tokens. The standard for user created apps running an API seems too be Oauth2. But, I don't think that is what RainMachine is doing. But, if the RainMachine syntax is more widely used, or there is a way to "tune" the syntax in the plugin, well, maybe.

Posted on
Mon Jun 11, 2018 7:27 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: New Plugin: GhostXML

berkinet wrote:
...I might just get inspired...

Rainy day and I got inspired to play around a bit with Token auth -- on a very limited scale. I did not check out a copy of the software, preferring, for the moment, to just play locally. This seems manageable since the scope of the changes is quite small.

First, I modified the device somewhat in the authentication section. It struck me the useAuth checkbox was redundant with the digsetType menu. So, I deleted the useAuth checkbox and renamed the menu item useAuth. Then I added two entries so the list now contains the values: none, basic, digest or token. I also did a visibleBinding of the authentication data (username, password and token URL) to the value of authType. I think this removes possible confusion.

That made it easy to modify getTheData so the auth type can now be read directly and the test for auth type becomes
Code: Select all
auth_type = dev.pluginProps.get('useAuth', "none")
...
if auth_type == 'digest':
...
elif auth_type == 'basic':
...
else:/code].  Since the token acquisition doesn't really use any authentication services I just stuck it in the [i]else[/i] giving [code]else:
                if auth_type == 'token':
                    # We need to get a token to get started
                    a_url = dev.pluginProps['tokenUrl']
                    curl_arg = "/usr/bin/curl -vsk -H 'Content-Type: application/json' -X POST --data-binary '{ \"pwd\": \"" + password + "\", \"remember\": 1 }' '} ' " + a_url

                    proc = subprocess.Popen(curl_arg, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

                    reply_in = proc.communicate()
                    reply = simplejson.loads(reply_in[0])
                    token = (reply["access_token"])

                    # Now, add the token to the end of the url
                    url = url + "?access_token=" + token
                proc = subprocess.Popen(["curl", '-vsk', url], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Note the single argument for curl passed to subprocess.Popen. This was because, for the life of me, I could not coerce subprocess.Popen to accept double-hyphen (I.e. GNU type) arguments as individual array elements and I needed to use --data-binary.

Posted on
Mon Jun 11, 2018 3:19 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

Thank you for this! Progress goes well on the refactoring (thanks to fellow developer RogueProeliator!)

I will definitely take this code forward for the next build. Cheers!

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Mon Jun 18, 2018 5:45 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

berkinet wrote:
...I might just get inspired...

Getting ready to fold this bit in to the latest build and I want to confirm my revisions to the code posted above (there were some formatting gremlins in the posted code). This is what I'm going to go with:

Code: Select all
auth_type = dev.pluginProps.get('useAuth', "none")
...
if auth_type == 'digest':
...
elif auth_type == 'basic':
...
else:
    if auth_type == 'token':
        # We need to get a token to get started
         a_url = dev.pluginProps['tokenUrl']
         curl_arg = "/usr/bin/curl -vsk -H 'Content-Type: application/json' -X POST --data-binary '{ \"pwd\": \"" + password + "\", \"remember\": 1 }' '} ' " + a_url

         proc = subprocess.Popen(curl_arg, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

         reply_in = proc.communicate()
         reply = simplejson.loads(reply_in[0])
         token = (reply["access_token"])

        # Now, add the token to the end of the url
         url = url + "?access_token=" + token
         proc = subprocess.Popen(["curl", '-vsk', url], stdout=subprocess.PIPE, stderr=subprocess.PIPE)


The above code is untested by me at this point. If I'm not mistaken, the auth_type == 'token' would need to be an elif with the else being auth_type == 'None' or something similar (as I believe there would be, in effect, four auth_types to support--Basic, Digest, Token, None).

Thanks again for the leg work on this!

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Mon Jun 18, 2018 10:47 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: New Plugin: GhostXML

DaveL17 wrote:
berkinet wrote:
...I might just get inspired...

Getting ready to fold this bit in to the latest build and I want to confirm my revisions to the code posted above (there were some formatting gremlins in the posted code). This is what I'm going to go with:...

I am away until Wednesday so I do not have an absolute answer. But, the code seems right. It just needs the corresponding changes in the device.xml. If I can access my computer from where I am, I will post that tonight. Otherwise, Wednesday
Last edited by berkinet on Mon Jun 18, 2018 10:55 am, edited 1 time in total.

Posted on
Mon Jun 18, 2018 10:55 am
berkinet offline
User avatar
Posts: 3290
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: New Plugin: GhostXML

I wrote:
...It just needs the corresponding changes in the device.xml. If I can access my computer from where I am, I will post that tonight...

Ok. Got it.

Code: Select all
 <Device type="custom" id="GhostXMLdevice">
        <Name>GhostXML Device</Name>

        <ConfigUI>
         ...
         ...
            <!-- Authentication -->

            <Field id="authTitle" type="label" alignText="right">
                <Label>Authentication</Label>
            </Field>

            <Field id="authSep" type="separator"/>

            <Field id="authLabel" type="label" fontSize="small">
                <Label>If your source requires authentication, tick the Use Authentication box, select type of authentication used, and enter the username and password.</Label>
            </Field>

<!--             <Field id="useAuth" type="checkbox" defaultValue="false">
                <Label>Use Authentication?</Label>
            </Field> -->

            <Field id="useAuth" type="menu" defaultValue="none" tooltip="Select the appropriate authentication type for your data source." >>
                <Label>Method:</Label>
                <List>
                    <Option value="none">None</Option>
                    <Option value="basic">Basic</Option>
                    <Option value="digest">Digest</Option>
                    <Option value="token">Token</Option>
                </List>
            </Field>

            <!-- Note: although the username and password fields are called
                 digestUser and digestPass, these fields are now used for both
                 digest and basic auth. -->
            <Field id="digestUser" type="textfield" tooltip="Enter or paste a Username ID here."
                visibleBindingId="useAuth" visibleBindingValue="basic, digest">
                <Label>Username:</Label>
            </Field>

            <Field id="digestPass" type="textfield" tooltip="Enter or paste a Password ID here."
                visibleBindingId="useAuth" visibleBindingValue="basic, digest, token">
                <Label>Password:</Label>
            </Field>

            <Field id="tokenUrl" type="textfield" tooltip="Enter the URL for the Token server."
                visibleBindingId="useAuth" visibleBindingValue="token">
                <Label>Token URL:</Label>
            </Field>

      ... ...
        </ConfigUI>

Posted on
Mon Jun 18, 2018 3:43 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

Thank you!

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Mon Jun 18, 2018 3:59 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

I had no idea that you could bind a control to more than one value. Learned something new!

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun Jul 08, 2018 8:55 pm
bsp9493 offline
Posts: 153
Joined: Nov 30, 2017
Location: Kelowna, BC

Re: New Plugin: GhostXML

Just updated to 4.06... current version I believe.

getting the following in my error logs on restart...

what did I do... no issues prior to upgrade.

Thanx in advance

dave



Jul 8, 2018, 7:52:31 PM
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev

Jul 8, 2018, 7:53:00 PM
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev

Jul 8, 2018, 7:53:30 PM
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev

Posted on
Mon Jul 09, 2018 5:28 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

Hi Dave - You did nothing to cause this.

In this latest build, I refactored a few things in the code that caused some things to be renamed under the hood. The quick fix is to go to your GhostXML Action Items, open each one in turn, and re-select the GhostXML action to perform. Under each Action Item, select:

Type --> Device Actions --> GhostXML Controls --> Refresh Data For Device

And you should be in business.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Mon Jul 09, 2018 9:37 am
bsp9493 offline
Posts: 153
Joined: Nov 30, 2017
Location: Kelowna, BC

Re: New Plugin: GhostXML

hmmm...

ok, maybe I have been using your plug in incorrectly.

Originally, I had installed the plugin, - configured to 30 second timeout and debug (for now)

I had then created 4 separate devices (to read xml files from my Autelis Pool controller - (pumps, chem, names, status) - refresh frequency currently set to 30 seconds, nothing else special.

This was working

I do NOT have any Action Groups or items set for these devices.

I have now created 4 action groups as you noted to correspond to the above names. I have executed each individually. Same issue :(

see debug log below.

GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev
GhostXML Debug User prefs saved.
GhostXML Debugging on (Level: Debug (10)
GhostXML Debug valuesDict: UiValuesDict : (dict)
configMenuServerTimeout : 30 (string)
debugHeaderSpace : (string)
debugLabel : (string)
infoLabel0 : (string)
oldDebugLevel : 20 (string)
sep2 : (string)
sep3 : (string)
sep4 : (string)
separator01 : (string)
showDebugInfo : false (bool)
showDebugLevel : 10 (integer)
space0 : (string)
space06 : (string)
space07 : (string)
space10 : (string)
space11 : (string)
space4 : (string)
space5 : (string)
space6 : (string)
space7 : (string)
space8 : (string)
space9 : (string)
updaterEmail : bsp9493@gmail.com (string)
updaterEmailsEnabled : true (bool)
updaterLastCheck : 1531080448 (real)
updaterLastVersionEmailed : 0.4.06 (string)
Action Group GhostXML Chem refresh
Action Group GhostXML Names refresh
Action Group GhostXML Pumps refresh
Action Group GhostXML Status refresh
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev

Posted on
Mon Jul 09, 2018 9:51 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: New Plugin: GhostXML

An Action Item wouldn't be required for "normal" use of the plugin. If regular (periodic) updates are all you need, then a standard GhostXML device is all that would be required. Set up the device to access your data source and have it refresh at some useful interval.

GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev
GhostXML Error plugin does not have a <CallbackMethod> specified for action id refreshDataForDev

These error messages are telling me that somewhere in your setup there are GhostXML actions that are calling the "old" version of the code (using the Action ID 'refreshDataForDev' which has been renamed to 'refresh_data_for_dev').

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Who is online

Users browsing this forum: No registered users and 1 guest