New Plugin: GhostXML

Hi, Having a bit of trouble on a couple of fronts. First on an older mac mini running 10.9.5 and Indigo 6.1.11 I get an error in 0.3.13:

Traceback (most recent call last):
SyntaxError: ('invalid syntax', ('plugin.py', 570, 35, " chars_to_remove = {'/', '(', ')'}\n"))

If I comment this out it works fine.

With Indigo 7.1.1, a newer mac mini and 10.13.2, the program runs very slow and fails to launch a dialog box. I get this error after trying to add a new device:
Error (client) timeout waiting for plugin response from com.fogbert.indigoplugin.GhostXML for request UiGetValues2
Error (client) timeout waiting for plugin response from com.fogbert.indigoplugin.GhostXML for request CallPluginFunc
Error (client) runDialogForDevice() caught exception: NSInvalidArgumentException -- *** -[__NSPlaceholderDictionary initWithObjects:forKeys:count:]: attempt to insert nil object from objects[0]

If I comment out the same line and then try and work on a second device, it is very fast and works fine.

The second and more pressing issue is that I have a JSON stream that I can link to using the auth and proper web address. An example output in the browser would be:

{"id":"dia/channel/00000000-00000000-00409DFF-FF5C532F/wine_closet/temperature","type":"DOUBLE","value":"55.6","units":"F","timestamp":"2017-12-23T16:44:20.000Z","server_timestamp":"2017-12-23T16:44:48.134Z","history_uri":"/ws/v1/streams/history/dia/channel/00000000-00000000-00409DFF-FF5C532F/wine_closet/temperature"}

To my mind this should show up with a key:value of:

id:dia/channel/00000000-00000000-00409DFF-FF5C532F/wine_closet/temperature
type:DOUBLE
value:55.6
units:F
timestamp:2017-12-23T16:44:20.000Z
server_timestamp:2017-12-23T16:44:48.134Z
history_uri:/ws/v1/streams/history/dia/channel/00000000-00000000-00409DFF-FF5C532F/wine_closet/temperature

what I get instead is:

GhostXML Error Digi json feed: No JSON object could be decoded
GhostXML Error Error cleaning dictionary keys: 'NoneType' object has no attribute 'iterkeys'

and the device dialog box shows:

DeviceOnline:True
devicelastudpated: "the correct date"
devicetimestamp: "the current timestamp"

Any ideas on what is going on on either of these fronts?

Thanks!

Here is a new wrinkle. If I save the output from the JSON stream as a local file, it works fine - see bottom image. If I do the JSON feed with the web address I get the first status image and a the the device shows a device error in indigo.

A bit stumped to say the least.
I've previously setup a python script that uses basic Auth and that works fine. It appears that it could be a problem of using auth digest vs. basic auth, but this goes beyond my knowledge if this makes sense. Any help would be appreciated.

Thanks,

Geoff

Here is the full url in the set up menu:

https://my.idigi.com/ws/v1/streams/inve ... ature.json

Hi Geoff - sorry you're having trouble. I was in the middle of a reply when your second message popped up. I was also able to read the JSON without error when it was saved locally and I'm pretty sure that it's an issue with authentication.

When you read the example JSON, did you read it with this URL in a browser?
https://my.idigi.com/ws/v1/streams/inventory/dia/channel/00000000-00000000-00409DFF-FF5C532F/wine_closet/temperature.json

Yes. If I have already logged in, I see the json and if I open a new browser, I'm prompted to enter user and password info. Pretty sure this is an auth problem at this point.

Looks like a Device Cloud resource, is that right?

Looking over their API documentation leads me to believe that the service requires OAUTH authentication and--regrettably--OAUTH is not supported by the plugin.

Hi Dave,

I think you are right. Is something that is supportable in the future? I'm far from any programming genius, but I have gotten this to work with a python script to load data into indigo. Is there naything I can do to either try and do this myself or help out?

Thanks,

Geoff

Went into the python code and just removed --digest and it now works. Perhaps there is an easy way to make a toggle button in the setup to select between digest and basic.

Thanks for the help and super cool plugin.

-Geoff

# proc = subprocess.Popen(["curl", '-vs', '--digest', '-u', username + ':' + password, url], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc = subprocess.Popen(["curl", '-vs', '-u', username + ':' + password, url], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Please do me a solid and turn off digest auth and then enter the url in the following format and let me know if it works for you:

Code: Select all

https://username:password@hostname/path

That totally works! I uncommented the --digest line and restored it to normal and set up the URL as you described. Nice. You probably have better things to do at the moment, but a more elegant error catch for this, a way to configure auth digest vs. auth basic and a way to secure the password rather than having it stored in plain test would be very helpful next time you are playing around and updating the code.

Thanks for the help!

Cheers,

Geoff

Glad to hear that's working for you. I have it on my to do list to change the way that authentication is handled within the configuration dialog to allow the username and password fields to work for both kinds of authentication. You are absolutely right that there needs to be a more elegant trap for instances where the wrong auth method is used.

The password thing is a bit trickier though.

• For basic authentication (what you're using here), the username and password are sent in the clear as a part of the http transmission anyway.
• Indigo stores configuration parameters in plain text within the settings file. The software does allow for a password field to be masked, but that only obscures the information on the display.
• Aside from instances where someone has gained direct access to your server (which itself should be password protected), I feel there's little to be gained by trying to hide that information.

I don't know of any plugin that has a facility to store hashed username/password information in a truly secure way (but that doesn't mean that one doesn't exist). To be honest, I'm not sure I want to take on that responsibility--but I'm open to ideas!

Enjoy the plugin!
Dave

Hello All,

I am new to Mac, I am New to Indigo and I am new to GhostXML, So please forgive me if I am doing something stupid.

I have installed the plugin and it was working great for a while and then I start to see these errors being logged in the event viewer.

GhostXML Error : Error in plugin execution ExecuteAction:
Traceback
File "plugin.py", Line 802, in refreshdataforDeviceAction
File "plugin.py", Line 737, In RefreshDataForDevice
AttributeError: 'PluginDevice' object has no attribute 'configured'

Any pointers would be great, Thanks
Deuan

Hello All,

I am new to Mac, I am New to Indigo and I am new to GhostXML, So please forgive me if I am doing something stupid.

I have installed the plugin and it was working great for a while and then I start to see these errors being logged in the event viewer.

GhostXML Error : Error in plugin execution ExecuteAction:
Traceback
File "plugin.py", Line 802, in refreshdataforDeviceAction
File "plugin.py", Line 737, In RefreshDataForDevice
AttributeError: 'PluginDevice' object has no attribute 'configured'

Any pointers would be great, Thanks
Deuan

Hi Deuan - welcome!

It sounds like your GhostXML device may be in a bad state. The plugin device should definitely have that attribute. Please do me a favor and take the following steps and post your results back here.

1. Go to Actions --> Create New Action Group --> Server Actions --> Script and File Actions --> Execute Script
2. In the window that opens, name the Action 'Python Scratchpad' or 'Temp" or whatever you like.
3. Paste the following script into the window and make sure that 'Embedded' is selected:

Code: Select all

dev = indigo.devices[1086913868]
indigo.server.log(unicode(dev))


4. Leave that window open, go back to the Indigo UI, and right-click on the GhostXML device in question and select Copy ID. Paste your device ID in place of the number 1086913868 within the script.
5. Click the run button.
6. Click OK to save the action or click cancel. (I recommend saving it to have a nice scratchpad to test and run Python scripts within the Indigo environment later on.)

The information that's written to the Indigo log window will help me understand what's going on. Be sure to review the information that you post to make sure it doesn't have any usernames, passwords, or other sensitive information. If you need help following these steps, please let me know. It can seem intimidating at first but it's not too bad.

Hi Dave,

Thanks for the quick reply.

Here is the info you requested.


Script address :
batteryLevel : None
buttonGroupCount : 0
configured : True
description :
deviceTypeId : GhostXMLdevice
displayStateId : deviceIsOnline
displayStateImageSel : SensorOn
displayStateValRaw : True
displayStateValUi : Enabled
enabled : True
energyAccumBaseTime : None
energyAccumTimeDelta : None
energyAccumTotal : None
energyCurLevel : None
errorState :
folderId : 538469380
globalProps : MetaProps : (dict)
com.fogbert.indigoplugin.GhostXML : (dict)
digestPass : (string)
digestUser : (string)
doSubs : false (bool)
feedType : JSON (string)
refreshFreq : 0 (string)
sourceXML : http://10.10.10.92/solar_api/v1/GetPowe ... eData.fcgi (string)
subA : solar (string)
subB : (string)
subC : (string)
subD : (string)
subE : (string)
useDigest : false (bool)
id : 866117516
lastChanged : 2018-01-05 09:16:59
lastSuccessfulComm : 2018-01-05 09:16:59
model : GhostXML Device
name : Solar
ownerProps : com.fogbert.indigoplugin.GhostXML : (dict)
digestPass : (string)
digestUser : (string)
doSubs : false (bool)
feedType : JSON (string)
refreshFreq : 0 (string)
sourceXML : http://10.10.10.92/solar_api/v1/GetPowe ... eData.fcgi (string)
subA : solar (string)
subB : (string)
subC : (string)
subD : (string)
subE : (string)
useDigest : false (bool)
pluginId : com.fogbert.indigoplugin.GhostXML
pluginProps : emptyDict : (dict)
protocol : Plugin
remoteDisplay : True
states : States : (dict)
Body_Data_Inverters_1_DT : 122 (string)
Body_Data_Inverters_1_E_Day : 34220 (string)
Body_Data_Inverters_1_E_Total : 4536499.5 (string)
Body_Data_Inverters_1_E_Year : 129445.601562 (string)
Body_Data_Inverters_1_P : None (string)
Body_Data_Site_E_Day : 34220 (string)
Body_Data_Site_E_Total : 4536499.5 (string)
Body_Data_Site_E_Year : 129445.601562 (string)
Body_Data_Site_Mode : vague-meter (string)
Body_Data_Site_P_Akku : None (string)
Body_Data_Site_P_Grid : None (string)
Body_Data_Site_P_Load : None (string)
Body_Data_Site_P_PV : None (string)
Body_Data_Site_rel_Autonomy : None (string)
Body_Data_Site_rel_SelfConsumption : None (string)
Head_Status_Code : 0 (string)
Head_Status_Reason : (string)
Head_Status_UserMessage : (string)
Head_Timestamp : 2018-01-04T22:40:21+11:00 (string)
deviceIsOnline : true (bool)
deviceIsOnline.ui : Enabled (string)
deviceLastUpdated : 01/04/2018 at 22:48 (string)
deviceTimestamp : 1515067086.858169 (string)
subModel :
supportsAllLightsOnOff : False
supportsAllOff : False
supportsStatusRequest : False
version : None

Kind Regards
Deuan

Perfect. So I see that the attribute 'configured' is there and set to True. So let's try a couple more things:

Go into the device's configuration dialog and then select save and see if that clears the error.

If that doesn't work, try creating a new GhostXML device and use this URL and see if that throws the same error:

Code: Select all

http://services.faa.gov/airport/status/ORD.xml

If both of these prove fruitless, please turn on debugging, reload the plugin and post the log output (including the bit at the start which has your plugin version and whatnot).

Hi Dave,

I have added your XML sample device and I don't see the same problem. The device I am reading in a solar system and the data is in json.

With that said, I think I have found the issue. The device "self" refresh has a minimum value of 15 seconds and I wanted to retrieve the data much faster, around 2 seconds. To achieve this, I set the device to manual and setup a schedule to refresh the data for that device every 2 seconds. However as soon as I use the Schedules to request the plugin to refresh the data for a device the error appears. If I simply leave with a 15 seconds on the device itself the error has gone.

I hope this helps you figure out where the issues is.

ps, With the XML test device I get the same error when forcing a refresh from the schedules.

Thanks for your help.
Deuan