New Plugin: GhostXML

I think I have a viable solution to the illegal character error. Using @Albatros' JSON as a source:

Code: Select all

Nov 25, 2016, 9:17:54 AM
   Interactive Shell               States : (dict)
     deviceIsOnline : true (bool)
     deviceIsOnline.ui : Enabled (string)
     deviceLastUpdated : 11/25/2016 at 09:14 (string)
     deviceTimestamp : 1480086858.736982 (real)
     request_route : /get-sensors (string)
     response_energylinks_0_aggregate_dayTotal : 37.94 (string)
     response_energylinks_0_aggregate_po : 4170 (string)
     response_energylinks_0_aggregate_po_minus_ : 330 (string)
     response_energylinks_0_aggregate_po_minus_t : 06:12 (string)
     response_energylinks_0_aggregate_po_plus_ : 5920 (string)
     response_energylinks_0_aggregate_po_plus_t : 10:04 (string)
     response_energylinks_0_c1 : 2000 (string)
     response_energylinks_0_c2 : 1000 (string)
     response_energylinks_0_code : 786373 (string)
     response_energylinks_0_favorite : no (string)
     response_energylinks_0_gas_dayTotal : 0.95 (string)
     response_energylinks_0_gas_lastHour : 0.0 (string)
     response_energylinks_0_id : 0 (string)
     response_energylinks_0_kwhindex : 2.16 (string)
     response_energylinks_0_name : EnergyLink (string)
     response_energylinks_0_s1_dayTotal : 9.18 (string)
     response_energylinks_0_s1_po : 694 (string)
     response_energylinks_0_s1_po_minus_ : 12 (string)
     response_energylinks_0_s1_po_minus_t : 00:01 (string)
     response_energylinks_0_s1_po_plus_ : 3719 (string)
     response_energylinks_0_s1_po_plus_t : 06:09 (string)
     response_energylinks_0_s2_dayTotal : 17.38 (string)
     response_energylinks_0_s2_po : 0 (string)
     response_energylinks_0_s2_po_minus_ : 0 (string)
     response_energylinks_0_s2_po_minus_t : 08:52 (string)
     response_energylinks_0_s2_po_plus_ : 3399 (string)
     response_energylinks_0_s2_po_plus_t : 08:06 (string)
     response_energylinks_0_t1 : solar (string)
     response_energylinks_0_t2 : other (string)
     response_energylinks_0_tariff : 2 (string)
     response_energylinks_0_used_dayTotal : 47.03 (string)
     response_energylinks_0_used_po : 4864 (string)
     response_energylinks_0_used_po_minus_ : 350 (string)
     response_energylinks_0_used_po_minus_t : 06:15 (string)
     response_energylinks_0_used_po_plus_ : 9066 (string)
     response_energylinks_0_used_po_plus_t : 10:04 (string)
     response_energylinks_0_wp : 4250 (string)
     response_preset : 2 (string)
     response_switches_0_favorite : no (string)
     response_switches_0_id : 0 (string)
     response_switches_0_mode : 0 (string)
     response_switches_0_name : Luxaflex groot (string)
     response_switches_0_type : somfy (string)
     response_switches_1_favorite : no (string)
     response_switches_1_id : 1 (string)
     response_switches_1_mode : 0 (string)
     response_switches_1_name : Rolluik Jesper  (string)
     response_switches_1_type : somfy (string)
     response_switches_2_favorite : no (string)
     response_switches_2_id : 40 (string)
     response_switches_2_mode : 0 (string)
     response_switches_2_name : Screen keuken (string)
     response_switches_2_type : somfy (string)
     response_switches_3_favorite : no (string)
     response_switches_3_id : 41 (string)
     response_switches_3_mode : 0 (string)
     response_switches_3_name : Rolluik Jesper (string)
     response_switches_3_type : somfy (string)
     response_switches_4_favorite : no (string)
     response_switches_4_id : 42 (string)
     response_switches_4_mode : 0 (string)
     response_switches_4_name : io (string)
     response_switches_4_type : somfy (string)
     response_switches_5_favorite : no (string)
     response_switches_5_id : 43 (string)
     response_switches_5_mode : 0 (string)
     response_switches_5_name : Luxaflex klein (string)
     response_switches_5_type : somfy (string)
     response_thermometers_0_channel : 1 (string)
     response_thermometers_0_favorite : no (string)
     response_thermometers_0_hu : 65 (string)
     response_thermometers_0_hu_minus_ : 63 (string)
     response_thermometers_0_hu_minus_t : 08:37 (string)
     response_thermometers_0_hu_plus_ : 66 (string)
     response_thermometers_0_hu_plus_t : 00:00 (string)
     response_thermometers_0_id : 0 (string)
     response_thermometers_0_model : 0 (string)
     response_thermometers_0_name : Slaapkamer TN (string)
     response_thermometers_0_outside : yes (string)
     response_thermometers_0_te : 16.5 (string)
     response_thermometers_0_te_minus_ : 16.3 (string)
     response_thermometers_0_te_minus_t : 09:23 (string)
     response_thermometers_0_te_plus_ : 16.7 (string)
     response_thermometers_0_te_plus_t : 00:00 (string)
     response_time : 2016-11-23 17:24 (string)
     status : ok (string)
     version : 3.372 (string)


And it looks proper. The new code contains a dict of replacement pairs as noted above and any new illegal characters can be easily added. If you agree, I'll push an update.
Dave

Is there any way to do authentication via this plugin in the URL request? I confess I'm nowhere near the programmer needed to figure this out. The ST community seems to think this is possible using curl: https://community.smartthings.com/t/jso ... -2/35866/4

ST does offer an XML version of all device statuses, which, combined with this plugin, is the holy grail of legacy support for me (and potentially, for using any ST hub as a Zigbee controller in Indigo!)

My problem is that most all ST documentation immediately dives deep and leaves me lost. Especially since it's aimed more at those writing apps for their hub, not those trying to send/receive data & commands directly to the hub.

Edit: looks like this is the actual documentation needed for external calls.

Is there any way to do authentication via this plugin in the URL request?

If your source will allow basic authentication, you should be able to include the username/password as a part of the URL with something like this which will send your username and password in clear text and is totally insecure.

Code: Select all

http://username:password@example.com/

However, if all communication is on your local LAN, that may not be a concern for you. Digest (secure) authentication is something else entirely. I have avoided authentication within the plugin because I don't want to take on responsibility for security. That said, I donated this plugin to the world, so if someone wants to take on digest authentication they could fork the plugin or issue a pull request.

Thanks Dave.

ST won't let me talk directly to the hub. (I know rite?) I have to talk to their cloud and use 0Auth authentication.

I'm beginning to think I'll have to do this from inside a python script, which probably means I won't be able to figure it out.

Dave - new trap makes sense - go ahead.

DC - I'll think about auth - the requests lib is now in I7 by default so ought to be doable.


Sent from my iPhone using Tapatalk

Ah - not touching oAuth yet!!


Sent from my iPhone using Tapatalk

New GhostXML Plugin Update

v0.3.03
- Adds an additional trap to address certain characters in key names that are not valid for Indigo state names.

You can download it here: GhostXML Plugin

YES, this is working....

Guys you are great!!!

For Dutch users with a Homewizard. With this pluging you can get all the energy data form you p1 port of your smart meter of your house. And also all your Homewizard devices. I am just a user who wants to inform but for a small amount you kan get energy/water data, somfy RTS and RF devices control. Controling homewizard devices is done with some small python code.

YES, this is working....

Guys you are great!!!

For Dutch users with a Homewizard. With this pluging you can get all the energy data form you p1 port of your smart meter of your house. And also all your Homewizard devices. I am just a user who wants to inform but for a small amount you kan get energy/water data, somfy RTS and RF devices control. Controling homewizard devices is done with some small python code.

Glad that this is working for you and I appreciate your willingness to share your experience with other users.

DaveL,

Thanks a lot for this plugin - as I mentioned above you cannot underestimate how flexible it is.

I have forked the github and have had a go at adding Digest auth (which I have working okay). Will copy updated code and issue pull request as soon as I can - for you to have a look at.

But also struggling with a Json seemingly flatdict issue: (which was there before and after minor Digest changes)

The below: Seems to fail within flatdict - with a empty {} result.

Code: Select all

[{"type":"PCU","devices":[{"part_num":"800-00356-r04","installed":"1470228526","serial_num":"REDACTED","device_status":["envoy.cond_flags.pcu_ctrl.commandedreset","envoy.cond_flags.pcu_ctrl.dc-pwr-low"],"last_rpt_date":"1470249596","admin_state":1,"created_date":"1470228526","img_load_date":"1424997903","img_pnum_running":"520-00045-r01-v01.22.00","ptpn":"540-00087-r01-v01.22.00","producing":true,"communicating":true,"chaneid":1627392273,"device_control":[{"gficlearset":false}]},{"part_num":"800-00356-r04","installed":"1470228529","serial_num":"REDACTED","device_status":["envoy.cond_flags.obs_strs.discovering"],"last_rpt_date":"1470249602","admin_state":1,"created_date":"1470228529","img_load_date":"1424997903","img_pnum_running":"520-00045-r01-v01.22.00","ptpn":"540-00087-r01-v01.22.00","producing":false,"communicating":true,"chaneid":1627392529,"device_control":[{"gficlearset":false}]},{"part_num":"800-00356-r04","installed":"1470228533","serial_num":"REDACTED","device_status":["envoy.global.ok"],"last_rpt_date":"1470249613","admin_state":1,"created_date":"1470228533","img_load_date":"1424997903","img_pnum_running":"520-00045-r01-v01.22.00","ptpn":"540-00087-r01-v01.22.00","producing":true,"communicating":true,"chaneid":1627392785,"device_control":[{"gficlearset":false}]}]}]


On fiddling with Flatdict - it appears to fail because of the starting list [ and ending ].
If I delete these - it works okay within flatdict - and would presume GhostXML - although haven't tested.

Could add check/or option for leading [ and ending ]?
Any thoughts?

Glenn

It looks like at least part of the problem is the fact that the dictionary is wrapped in a list. I also needed to change true/false to True/False and then was able to do this:

Code: Select all

import flatdict

values = {"type":"PCU","devices":[{"part_num":"800-00356-r04","installed":"1470228526","serial_num":"REDACTED","device_status":["envoy.cond_flags.pcu_ctrl.commandedreset","envoy.cond_flags.pcu_ctrl.dc-pwr-low"],"last_rpt_date":"1470249596","admin_state":1,"created_date":"1470228526","img_load_date":"1424997903","img_pnum_running":"520-00045-r01-v01.22.00","ptpn":"540-00087-r01-v01.22.00","producing":True,"communicating":True,"chaneid":1627392273,"device_control":[{"gficlearset":False}]},{"part_num":"800-00356-r04","installed":"1470228529","serial_num":"REDACTED","device_status":["envoy.cond_flags.obs_strs.discovering"],"last_rpt_date":"1470249602","admin_state":1,"created_date":"1470228529","img_load_date":"1424997903","img_pnum_running":"520-00045-r01-v01.22.00","ptpn":"540-00087-r01-v01.22.00","producing":False,"communicating":True,"chaneid":1627392529,"device_control":[{"gficlearset":False}]},{"part_num":"800-00356-r04","installed":"1470228533","serial_num":"REDACTED","device_status":["envoy.global.ok"],"last_rpt_date":"1470249613","admin_state":1,"created_date":"1470228533","img_load_date":"1424997903","img_pnum_running":"520-00045-r01-v01.22.00","ptpn":"540-00087-r01-v01.22.00","producing":True,"communicating":True,"chaneid":1627392785,"device_control":[{"gficlearset":False}]}]}

flat = flatdict.FlatDict(values)

indigo.server.log(str(flat))


and the result is:

Code: Select all

Nov 27, 2016, 7:31:42 PM
   Script                          {'devices:1:created_date': '1470228529', 'devices:0:device_control:0:gficlearset': False, 'devices:0:img_pnum_running': '520-00045-r01-v01.22.00', 'devices:0:communicating': True, 'devices:2:installed': '1470228533', 'devices:2:ptpn': '540-00087-r01-v01.22.00', 'devices:2:chaneid': 1627392785, 'devices:2:producing': True, 'devices:0:device_status:1': 'envoy.cond_flags.pcu_ctrl.dc-pwr-low', 'devices:0:device_status:0': 'envoy.cond_flags.pcu_ctrl.commandedreset', 'devices:1:last_rpt_date': '1470249602', 'devices:1:producing': False, 'devices:2:device_status:0': 'envoy.global.ok', 'devices:0:part_num': '800-00356-r04', 'devices:0:installed': '1470228526', 'devices:1:part_num': '800-00356-r04', 'devices:2:communicating': True, 'devices:0:producing': True, 'type': 'PCU', 'devices:0:serial_num': 'REDACTED', 'devices:2:admin_state': 1, 'devices:2:created_date': '1470228533', 'devices:1:img_pnum_running': '520-00045-r01-v01.22.00', 'devices:2:img_pnum_running': '520-00045-r01-v01.22.00', 'devices:1:serial_num': 'REDACTED', 'devices:0:ptpn': '540-00087-r01-v01.22.00', 'devices:1:chaneid': 1627392529, 'devices:2:part_num': '800-00356-r04', 'devices:1:device_control:0:gficlearset': False, 'devices:1:device_status:0': 'envoy.cond_flags.obs_strs.discovering', 'devices:1:communicating': True, 'devices:2:img_load_date': '1424997903', 'devices:1:img_load_date': '1424997903', 'devices:2:device_control:0:gficlearset': False, 'devices:2:serial_num': 'REDACTED', 'devices:1:installed': '1470228529', 'devices:2:last_rpt_date': '1470249613', 'devices:0:img_load_date': '1424997903', 'devices:0:created_date': '1470228526', 'devices:0:chaneid': 1627392273, 'devices:0:admin_state': 1, 'devices:1:ptpn': '540-00087-r01-v01.22.00', 'devices:0:last_rpt_date': '1470249596', 'devices:1:admin_state': 1}


Yep
Made the same discovery with the true - and then removal of starting and ending [] - gives a very usable response from flatdict (same as yours)
"(should be easy to add the true to True within substitute code)

If there are leading/ending [ ] shall they just be removed?

Although this is the other output trying to get functional:

Code: Select all

[{"serialNumber":"121630050316","lastReportDate":1480302511,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046735","lastReportDate":1480302509,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121632058899","lastReportDate":1480302472,"devType":1,"lastReportWatts":227,"maxReportWatts":270},{"serialNumber":"121630046772","lastReportDate":1480302474,"devType":1,"lastReportWatts":224,"maxReportWatts":270},{"serialNumber":"121630046255","lastReportDate":1480302478,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046238","lastReportDate":1480302485,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046227","lastReportDate":1480302481,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630050703","lastReportDate":1480302485,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630050397","lastReportDate":1480302492,"devType":1,"lastReportWatts":228,"maxReportWatts":270},{"serialNumber":"121630050272","lastReportDate":1480302489,"devType":1,"lastReportWatts":223,"maxReportWatts":270},{"serialNumber":"121630046782","lastReportDate":1480302499,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046206","lastReportDate":1480302507,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046661","lastReportDate":1480302514,"devType":1,"lastReportWatts":264,"maxReportWatts":270},{"serialNumber":"121630050417","lastReportDate":1480302514,"devType":1,"lastReportWatts":269,"maxReportWatts":271},{"serialNumber":"121630044744","lastReportDate":1480302446,"devType":1,"lastReportWatts":268,"maxReportWatts":270},{"serialNumber":"121630044459","lastReportDate":1480302453,"devType":1,"lastReportWatts":268,"maxReportWatts":270},{"serialNumber":"121630046708","lastReportDate":1480302458,"devType":1,"lastReportWatts":268,"maxReportWatts":270},{"serialNumber":"121630050211","lastReportDate":1480302463,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630050381","lastReportDate":1480302469,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046741","lastReportDate":1480302474,"devType":1,"lastReportWatts":225,"maxReportWatts":270},{"serialNumber":"121630046107","lastReportDate":1480302470,"devType":1,"lastReportWatts":265,"maxReportWatts":270},{"serialNumber":"121630050251","lastReportDate":1480302468,"devType":1,"lastReportWatts":270,"maxReportWatts":270},{"serialNumber":"121630046501","lastReportDate":1480302467,"devType":1,"lastReportWatts":268,"maxReportWatts":270},{"serialNumber":"121630046862","lastReportDate":1480302466,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046258","lastReportDate":1480302461,"devType":1,"lastReportWatts":265,"maxReportWatts":271},{"serialNumber":"121630046479","lastReportDate":1480302461,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630050383","lastReportDate":1480302459,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046432","lastReportDate":1480302457,"devType":1,"lastReportWatts":226,"maxReportWatts":270},{"serialNumber":"121630046250","lastReportDate":1480302455,"devType":1,"lastReportWatts":263,"maxReportWatts":271},{"serialNumber":"121630046308","lastReportDate":1480302452,"devType":1,"lastReportWatts":265,"maxReportWatts":271},{"serialNumber":"121630044455","lastReportDate":1480302450,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046485","lastReportDate":1480302448,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046769","lastReportDate":1480302447,"devType":1,"lastReportWatts":226,"maxReportWatts":270},{"serialNumber":"121630044415","lastReportDate":1480302515,"devType":1,"lastReportWatts":223,"maxReportWatts":270},{"serialNumber":"121630046779","lastReportDate":1480302518,"devType":1,"lastReportWatts":268,"maxReportWatts":270},{"serialNumber":"121632058618","lastReportDate":1480302518,"devType":1,"lastReportWatts":227,"maxReportWatts":270},{"serialNumber":"121630046205","lastReportDate":1480302476,"devType":1,"lastReportWatts":268,"maxReportWatts":270},{"serialNumber":"121630050349","lastReportDate":1480302482,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630050337","lastReportDate":1480302487,"devType":1,"lastReportWatts":223,"maxReportWatts":270},{"serialNumber":"121630046247","lastReportDate":1480302499,"devType":1,"lastReportWatts":261,"maxReportWatts":270},{"serialNumber":"121630046662","lastReportDate":1480302495,"devType":1,"lastReportWatts":261,"maxReportWatts":271},{"serialNumber":"121630046823","lastReportDate":1480302502,"devType":1,"lastReportWatts":221,"maxReportWatts":270},{"serialNumber":"121630046658","lastReportDate":1480302510,"devType":1,"lastReportWatts":259,"maxReportWatts":270},{"serialNumber":"121630050271","lastReportDate":1480302505,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046257","lastReportDate":1480302503,"devType":1,"lastReportWatts":262,"maxReportWatts":270},{"serialNumber":"121630050298","lastReportDate":1480302494,"devType":1,"lastReportWatts":225,"maxReportWatts":270},{"serialNumber":"121630050387","lastReportDate":1480302476,"devType":1,"lastReportWatts":269,"maxReportWatts":270},{"serialNumber":"121630046801","lastReportDate":1480302483,"devType":1,"lastReportWatts":222,"maxReportWatts":270},{"serialNumber":"121630046642","lastReportDate":1480302489,"devType":1,"lastReportWatts":222,"maxReportWatts":270},{"serialNumber":"121630046797","lastReportDate":1480302497,"devType":1,"lastReportWatts":269,"maxReportWatts":270}]


Again list within dict issue - but if I just remove [] dont have the same luck with example as above - with flatdict still showing as empty

Glenn

I'd say it's bad form to simply remove the leading/ending [], but it's easy to just extract what you need.

Code: Select all

>>> a = [{'a': 1, 'b': 2}]
>>> b = a[0]
>>> print(b)
{'a': 1, 'b': 2}


I'd say it's bad form to simply remove the leading/ending [], but it's easy to just extract what you need.

Code: Select all

>>> a = [{'a': 1, 'b': 2}]
>>> b = a[0]
>>> print(b)
{'a': 1, 'b': 2}


I agree - its bad form - but again the idea is to have something that will work with a whole range of inputs.

Could

Code: Select all

import types
if type(x) is list:
    print 'a list'
elif type(x) is tuple:
    print 'a tuple'
else:
    print 'neither a tuple or a list'


To check if list or tuple and then run through all items joining.

But again - starting to get a bit complicated to deal with various outputs.

Pity flatdict doesn't deal with lists - and there doesn't seem to be a better option. (did seen a flatten within Panda - which I'll check out next)

Glenn

I spent a fair bit of time searching available options and flatDict seemed to be the best available option.

The plugin is for the community to develop together, but my own personal preference would be to avoid expanding it to cover what might be one-off solutions (admittedly, a json array is probably not a one-off solution) when flatDict is expecting a json object. It wouldn't be too tough for a user to create a script to snag a copy of the data, convert it to a json object and save it to a local file--which could then be easily read by the plugin.