Indigo Domotics

Software Support and Home Automation Forum
https://forums.indigodomo.com/

New Plugin: GhostXML

https://forums.indigodomo.com/viewtopic.php?f=249&t=15137

Page 15 of 20

Re: New Plugin: GhostXML

PostPosted: Mon Mar 26, 2018 7:49 pm
by DaveL17
Scratch that. There was a bug with new devices when using basic auth (and maybe digest, too). I've uploaded v0.3.17 to the Indigo Plugin Store which should fix the problem. Please download and try that version instead.

I still believe that basic auth is what you're after.

Re: New Plugin: GhostXML

PostPosted: Tue Mar 27, 2018 9:04 pm
by hamw
Gave it a try but still no joy. Here is the log:

Code: Select all
   GhostXML                        There was a parse error. Check XML source.
   GhostXML Debug                  Device needs updating set to: True
   GhostXML Debug                  getDeviceStateList() method called.
   GhostXML Debug                  Pulling down existing state list.
   GhostXML Debug                    Writing dynamic states to device.
   GhostXML Debug                  No new states found.
   GhostXML Debug                  Device needs updating set to: False
   GhostXML Debug                  parseStateValues() method called.
   GhostXML Debug                  Writing device states:
   GhostXML Debug                     Response = Parse error. Check XML source.
   GhostXML Debug                  Digital Loggers Switch updated.

Mar 27, 2018, 10:59:02 PM
   GhostXML Debug                  Time since update (17) is greater than configured frequency (15)
   GhostXML Debug                  Queue task 409167487 sent to thread 409167487 - Digital Loggers Switch
   GhostXML Debug                  refreshDataForDev() method called.
   GhostXML Debug                  Found configured device: Digital Loggers Switch
   GhostXML Debug                  Refreshing device: Digital Loggers Switch
   GhostXML Debug                  getTheData() method called.
   GhostXML Debug                  *   Trying 10.0.1.XX...
* TCP_NODELAY set
* Connected to 10.0.1.XX (10.0.1.XX) port XXXX (#0)
* Server auth using Basic with user 'admin'
> GET /restapi/relay/outlets/all/state/=name,physical_state/ HTTP/1.1
> Host: 10.0.1.XX:XXXX
> Authorization: Basic YWRtaW46NDU2Q2FtM3JhWg==
> User-Agent: curl/7.51.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: close
< Content-Type: text/html
< Cache-Control: No-cache, no-store, must-revalidate, max-age=0
< Expires: Mon, 01 Jan 1990 00:00:01 GMT
< Pragma: no-cache
< Set-Cookie: DLILPC=""; Version=1; Max-Age=0; Path=/
<
{ [99 bytes data]
* Recv failure: Connection reset by peer
* Curl_http_done: called premature == 1
* Closing connection 0

   GhostXML Debug                  Source file type: XML
   GhostXML Debug                  stripNamespace() method called.
   GhostXML Debug                  <HTML><HEAD>

<META HTTP-EQUIV="refresh" content="0; URL=/index.htm">

</HEAD><BODY></BODY>
</HTML>
   GhostXML                        There was a parse error. Check XML source.
   GhostXML Debug                  Device needs updating set to: True


Other ideas? Does the request need to be reconfigured?

Re: New Plugin: GhostXML

PostPosted: Wed Mar 28, 2018 3:52 am
by DaveL17
Looks like you're on the right track and this line is promising:
Code: Select all
< HTTP/1.0 200 OK


What happens if you put the URL structure from your AppleScript directly into a web browser?
Code: Select all
http://usesr:pwd@10.0.1.XX:XXXX/outlet?

Re: New Plugin: GhostXML

PostPosted: Thu Mar 29, 2018 6:13 pm
by hamw
Well, when simply put in a web browser it returns the login screen, with the username and PW boxes. Maybe that's where it is getting hung up. But, it seems like the instructions would let you use the RestAPI to pull the info? I also put the Rest string in there and had the same result.

Re: New Plugin: GhostXML

PostPosted: Thu Mar 29, 2018 7:24 pm
by DaveL17
Based on this... Try setting up a device with digest auth and the url

Code: Select all
http://192.168.0.123/restapi/


Of course replacing the IP address with that of your device.

Re: New Plugin: GhostXML

PostPosted: Sat Apr 28, 2018 6:07 am
by hamw
I did as suggested and it made a slight difference:

Code: Select all
* TCP_NODELAY set
* Connected to 10.0.1.45 (10.0.1.45) port 8200 (#0)
* Server auth using Basic with user 'admin'
> GET /restapi HTTP/1.1
> Host: 10.0.1.45:8200
> Authorization: Basic YWRtaW46NDU2Q2FtM3JhWg==
> User-Agent: curl/7.51.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: close
< Content-Type: text/html
< Cache-Control: No-cache, no-store, must-revalidate, max-age=0
< Expires: Mon, 01 Jan 1990 00:00:01 GMT
< Pragma: no-cache
< Set-Cookie: DLILPC=""; Version=1; Max-Age=0; Path=/
<
{ [99 bytes data]
* Curl_http_done: called premature == 0                       <<<<<<<
* Closing connection 0

   GhostXML Debug                  Source file type: XML
   GhostXML Debug                  stripNamespace() method called.
   GhostXML Debug                  <HTML><HEAD>


At this line "called premature" formerly == 1

Progress?

Re: New Plugin: GhostXML

PostPosted: Sat Apr 28, 2018 7:22 am
by DaveL17
Definitely progress, but from the return, it looks like you're using basic auth instead of digest auth. Do you have the device set to use digest where the server is returning a message that basic is being used?

Re: New Plugin: GhostXML

PostPosted: Thu May 03, 2018 6:12 am
by qcvictor
Hi DaveL17
Have some trouble with json import, device stay stuck on "Processing" and no refresh anymore,
This the json output:
{"System":{
"Build":20102,
"Git Build":"mega-20180429",
"Local time":"2018-05-03 08:05:53",
"Unit":7,
"Name":"Wemos Pool Controller",
"Uptime":4671,
"Load":26,
"Load LC":11282,
"Free RAM":17912
},
"WiFi":{
"Hostname":"Wemos-Pool-Controller-7",
"IP config":"DHCP",
"IP":"10.0.0.XX",
"Subnet Mask":"255.255.255.0",
"Gateway IP":"10.0.X.X",
"MAC address":"5C:CF:7F:F0:28:8A",
"DNS 1":"10.0.X.X",
"DNS 2":"X.X.X.X",
"SSID":"XXXXXXXX",
"BSSID":"F0:9F:C2:A4:AE:EF",
"Channel":1,
"Connected msec":136965665,
"Last Disconnect Reason":201,
"Last Disconnect Reason str":"(201) No AP found",
"Number reconnects":26,
"RSSI":-53
},
"Sensors":[
{
"TaskNumber":1,
"Type":"Analog input - ADS1115",
"TaskName":"pH",
"TaskValues": [
{"ValueNumber":1,
"Name":"pH",
"Value":11.50}]
},
{
"TaskNumber":2,
"Type":"Analog input - internal",
"TaskName":"Pressure Filter",
"TaskValues": [
{"ValueNumber":1,
"Name":"Bar",
"Value":0.91}]
},
{
"TaskNumber":3,
"Type":"Display - LCD2004",
"TaskName":"Screen"},
{
"TaskNumber":4,
"Type":"Analog input - ADS1115",
"TaskName":"Orp",
"TaskValues": [
{"ValueNumber":1,
"Name":"Orp",
"Value":171.51}]
},
{
"TaskNumber":5,
"Type":"Switch input - Switch",
"TaskName":"PIR",
"TaskValues": [
{"ValueNumber":1,
"Name":"Switch",
"Value":0}]
},
{
"TaskNumber":6,
"Type":"Environment - DS18b20",
"TaskName":"Temperature",
"TaskValues": [
{"ValueNumber":1,
"Name":"Temperature",
"Value":9.06}]
},
{
"TaskNumber":7,
"Type":"Generic - Pulse counter",
"TaskName":"Water Flow",
"TaskValues": [
{"ValueNumber":1,
"Name":"Count",
"Value":0.00},
{"ValueNumber":2,
"Name":"Total",
"Value":0.00},
{"ValueNumber":3,
"Name":"Time",
"Value":0.00}]
},
{
"TaskNumber":9,
"Type":"Generic - Dummy Device",
"TaskName":"Test",
"TaskValues": [
{"ValueNumber":1,
"Name":"[Pressure Filter#Bar]+10",
"Value":0.00},
{"ValueNumber":2,
"Name":"Pressure Filter#Bar +10",
"Value":0.00},
{"ValueNumber":3,
"Name":"",
"Value":0.00},
{"ValueNumber":4,
"Name":"",
"Value":0.00}]
}
]
}

Output from debug:
Code: Select all
3 mai 2018 07:58:44
   GhostXML                        Debugging on.
   GhostXML Debug                  Debug level: 1

3 mai 2018 07:58:59
   GhostXML Debug                  Device has been updated. Blow state list up to Trigger and Control Page labels.
   GhostXML Debug                  validateDeviceConfigUi() method called.
   GhostXML Debug                  Device has been updated. Blow state list up to Trigger and Control Page labels.
   GhostXML Debug                  Stopping GhostXML device: ESP8266 Pool Level (json)
   GhostXML Debug                  Starting GhostXML device: ESP8266 Pool Level (json)
   GhostXML Debug                  Device has been updated. Blow state list up to Trigger and Control Page labels.
   GhostXML Debug                  Time since update (695) is greater than configured frequency (15)
   GhostXML Debug                  Found configured device: ESP8266 Pool Level (json)
   GhostXML Debug                  Refreshing device: ESP8266 Pool Level (json)
   GhostXML Debug                  *   Trying 10.0.0.56...
* TCP_NODELAY set
* Connected to 10.0.0.56 (10.0.0.56) port 80 (#0)
> GET /json HTTP/1.1
> Host: 10.0.0.56
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 242
< Connection: close
< Access-Control-Allow-Origin: *
<
{ [242 bytes data]
* Closing connection 0

   GhostXML Debug                  Source file type: JSON
   GhostXML Debug                  Device needs updating set to: True
   GhostXML Debug                  Pulling down existing state list.
   GhostXML Debug                    Writing dynamic states to device.
   GhostXML Debug                  Device needs updating set to: False

Re: New Plugin: GhostXML

PostPosted: Thu May 03, 2018 11:57 am
by DaveL17
Took me a while, but I figured out what the issue is.

Some of the source json keys contain spaces (which I don't believe is good form) but when I updated the plugin to contend with the spaces, it then parses successfully. It's a wonder that this particular problem has taken this long to surface.

I'll post an update to the plugin later tonight (North America time). You should then be in business.

Re: New Plugin: GhostXML

PostPosted: Thu May 03, 2018 6:16 pm
by DaveL17
I just released a new version of the plugin to the Indigo Plugin Store that should be able to handle your JSON feed.

Re: New Plugin: GhostXML

PostPosted: Fri May 04, 2018 3:35 am
by qcvictor
Hi Dave
Thanks you so much for your great job and support, it's working now.
Vic

Re: New Plugin: GhostXML

PostPosted: Fri May 04, 2018 3:54 am
by DaveL17
Excellent. Thanks for letting me know.

Interesting problem

PostPosted: Fri Jun 08, 2018 1:25 pm
by berkinet
I tried the GhostXML plugin to read data from my RainMachine irrigation controller using their RESTful API. If I receive the JSON data from the RainMachine and save it into a file, the plugin reads the file and updates the pseudo device as expected -- very cool, thanks!. However, if I have the plugin connect directly to the RainMaching using the same URL that works in Safari
Code: Select all
https://rainmachine.local:8080/api/4/watering/past
Here is the log with debugging set to high
    GhostXML Debug Refresh frequency: 0 (Manual refresh only)
    GhostXML Debug refreshData() method called.
    GhostXML Debug refreshDataForDev() method called.
    GhostXML Debug Found configured device: new device
    GhostXML Debug Refreshing device: new device
    GhostXML Debug getTheData() method called.
    GhostXML Debug * Trying 192.168.5.45...
    * TCP_NODELAY set
    * Connected to 192.168.5.45 (192.168.5.45) port 8080 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
    * successfully set certificate verify locations:
    * CAfile: /etc/ssl/cert.pem
    CApath: none
    * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    } [512 bytes data]
    * TLSv1.2 (IN), TLS handshake, Server hello (2):
    { [89 bytes data]
    * TLSv1.2 (IN), TLS handshake, Certificate (11):
    { [957 bytes data]
    * TLSv1.2 (OUT), TLS alert, Server hello (2):
    } [2 bytes data]
    * SSL certificate problem: self signed certificate
    * stopped the pause stream!
    * Closing connection 0

    GhostXML Debug Source file type: JSON
    GhostXML Debug parseTheJSON() method called.
    GhostXML Error new device: No JSON object could be decoded
    GhostXML Debug cleanTheKeys() method called.
    GhostXML Error Error cleaning dictionary keys: 'NoneType' object has no attribute 'iterkeys'
    GhostXML Debug Refresh frequency: 0 (Manual refresh only)
And, here is the JSON data returned
Code: Select all
{"pastValues": [{"pid": 7, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 8, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 3, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.464572714257837, "qpf": 7.9099999999999993}, {"pid": 2, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 11, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 1, "dateTimestamp": 1528408800, "dateTime": "2018-06-08 00:00:00", "used": true, "et0": 13.567305490819034, "qpf": 3.1000000000000001}]}
The data read from the file is identical.

For completeness, here is what I get back from calling the same url in curl[, and which is then saved to file, minus the headers.
Code: Select all
HTTP/1.1 200 OK
Content-type: text/html
Content-Length: 881
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: *
X-Frame-Options: DENY
Date: Fri, 08 Jun 2018 08:14:18 GMT
Server: lighttpd/1.4.31

{"pastValues": [{"pid": 7, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 8, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 3, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.464572714257837, "qpf": 7.9099999999999993}, {"pid": 2, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 11, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 1, "dateTimestamp": 1528408800, "dateTime": "2018-06-08 00:00:00", "used": true, "et0": 13.567305490819034, "qpf": 3.1000000000000001}]}+ exit 0

Re: New Plugin: GhostXML

PostPosted: Fri Jun 08, 2018 2:31 pm
by DaveL17
Thanks for reporting the error. I see the same one from time to time in my development environment on both URL-based as well as file-based sources. That said, the error isn't persistent and the device will right itself during a subsequent update. I believe the error is a result of code that isn't as thread-safe as it could be. I've been working on a fundamental refactoring which I hope will be considerably more stable and fully thread-safe that should address the issue. A couple of questions:

Does the error occur every time the device tries to update from the URL or only sometimes?
Do you have more than one GhostXML device configured?

Re: New Plugin: GhostXML

PostPosted: Fri Jun 08, 2018 2:48 pm
by berkinet
DaveL17 wrote:
...Does the error occur every time the device tries to update from the URL or only sometimes? Do you have more than one GhostXML device configured?

Well, I let it run for a while with 15 sec updates and it never connected. I have 2 devices, But, had the file based device turned off while this was happening (Indigo device comms unchecked). FWIW, the file based device never failed.

I have one additional note. Initially I created the file based device, then I copied it and modified it to use a URL. Since I copied another device the state’s were already populated. That URL based device never worked, though it continued to display the states it had inherited . I then deleted it and created a new device from scratch, which continued to have the same connection problem but, only displayed the meta device states like last time connected.
All times are UTC - 6 hours
Page 15 of 20
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/