New Plugin: GhostXML

Posted on
Mon Mar 26, 2018 7:49 pm
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

Scratch that. There was a bug with new devices when using basic auth (and maybe digest, too). I've uploaded v0.3.17 to the Indigo Plugin Store which should fix the problem. Please download and try that version instead.

I still believe that basic auth is what you're after.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue Mar 27, 2018 9:04 pm
hamw offline
Posts: 1107
Joined: Mar 31, 2008

Re: New Plugin: GhostXML

Gave it a try but still no joy. Here is the log:

Code: Select all
   GhostXML                        There was a parse error. Check XML source.
   GhostXML Debug                  Device needs updating set to: True
   GhostXML Debug                  getDeviceStateList() method called.
   GhostXML Debug                  Pulling down existing state list.
   GhostXML Debug                    Writing dynamic states to device.
   GhostXML Debug                  No new states found.
   GhostXML Debug                  Device needs updating set to: False
   GhostXML Debug                  parseStateValues() method called.
   GhostXML Debug                  Writing device states:
   GhostXML Debug                     Response = Parse error. Check XML source.
   GhostXML Debug                  Digital Loggers Switch updated.

Mar 27, 2018, 10:59:02 PM
   GhostXML Debug                  Time since update (17) is greater than configured frequency (15)
   GhostXML Debug                  Queue task 409167487 sent to thread 409167487 - Digital Loggers Switch
   GhostXML Debug                  refreshDataForDev() method called.
   GhostXML Debug                  Found configured device: Digital Loggers Switch
   GhostXML Debug                  Refreshing device: Digital Loggers Switch
   GhostXML Debug                  getTheData() method called.
   GhostXML Debug                  *   Trying 10.0.1.XX...
* TCP_NODELAY set
* Connected to 10.0.1.XX (10.0.1.XX) port XXXX (#0)
* Server auth using Basic with user 'admin'
> GET /restapi/relay/outlets/all/state/=name,physical_state/ HTTP/1.1
> Host: 10.0.1.XX:XXXX
> Authorization: Basic YWRtaW46NDU2Q2FtM3JhWg==
> User-Agent: curl/7.51.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: close
< Content-Type: text/html
< Cache-Control: No-cache, no-store, must-revalidate, max-age=0
< Expires: Mon, 01 Jan 1990 00:00:01 GMT
< Pragma: no-cache
< Set-Cookie: DLILPC=""; Version=1; Max-Age=0; Path=/
<
{ [99 bytes data]
* Recv failure: Connection reset by peer
* Curl_http_done: called premature == 1
* Closing connection 0

   GhostXML Debug                  Source file type: XML
   GhostXML Debug                  stripNamespace() method called.
   GhostXML Debug                  <HTML><HEAD>

<META HTTP-EQUIV="refresh" content="0; URL=/index.htm">

</HEAD><BODY></BODY>
</HTML>
   GhostXML                        There was a parse error. Check XML source.
   GhostXML Debug                  Device needs updating set to: True


Other ideas? Does the request need to be reconfigured?

Posted on
Wed Mar 28, 2018 3:52 am
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

Looks like you're on the right track and this line is promising:
Code: Select all
< HTTP/1.0 200 OK


What happens if you put the URL structure from your AppleScript directly into a web browser?
Code: Select all
http://usesr:pwd@10.0.1.XX:XXXX/outlet?

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Thu Mar 29, 2018 6:13 pm
hamw offline
Posts: 1107
Joined: Mar 31, 2008

Re: New Plugin: GhostXML

Well, when simply put in a web browser it returns the login screen, with the username and PW boxes. Maybe that's where it is getting hung up. But, it seems like the instructions would let you use the RestAPI to pull the info? I also put the Rest string in there and had the same result.

Posted on
Thu Mar 29, 2018 7:24 pm
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

Based on this... Try setting up a device with digest auth and the url

Code: Select all
http://192.168.0.123/restapi/


Of course replacing the IP address with that of your device.
Attachments
Screen Shot 2018-03-29 at 8.06.56 PM.png
Screen Shot 2018-03-29 at 8.06.56 PM.png (92.81 KiB) Viewed 952 times

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Apr 28, 2018 6:07 am
hamw offline
Posts: 1107
Joined: Mar 31, 2008

Re: New Plugin: GhostXML

I did as suggested and it made a slight difference:

Code: Select all
* TCP_NODELAY set
* Connected to 10.0.1.45 (10.0.1.45) port 8200 (#0)
* Server auth using Basic with user 'admin'
> GET /restapi HTTP/1.1
> Host: 10.0.1.45:8200
> Authorization: Basic YWRtaW46NDU2Q2FtM3JhWg==
> User-Agent: curl/7.51.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: close
< Content-Type: text/html
< Cache-Control: No-cache, no-store, must-revalidate, max-age=0
< Expires: Mon, 01 Jan 1990 00:00:01 GMT
< Pragma: no-cache
< Set-Cookie: DLILPC=""; Version=1; Max-Age=0; Path=/
<
{ [99 bytes data]
* Curl_http_done: called premature == 0                       <<<<<<<
* Closing connection 0

   GhostXML Debug                  Source file type: XML
   GhostXML Debug                  stripNamespace() method called.
   GhostXML Debug                  <HTML><HEAD>


At this line "called premature" formerly == 1

Progress?

Posted on
Sat Apr 28, 2018 7:22 am
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

Definitely progress, but from the return, it looks like you're using basic auth instead of digest auth. Do you have the device set to use digest where the server is returning a message that basic is being used?

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Thu May 03, 2018 6:12 am
qcvictor offline
Posts: 74
Joined: Apr 23, 2015
Location: Quebec, Canada

Re: New Plugin: GhostXML

Hi DaveL17
Have some trouble with json import, device stay stuck on "Processing" and no refresh anymore,
This the json output:
{"System":{
"Build":20102,
"Git Build":"mega-20180429",
"Local time":"2018-05-03 08:05:53",
"Unit":7,
"Name":"Wemos Pool Controller",
"Uptime":4671,
"Load":26,
"Load LC":11282,
"Free RAM":17912
},
"WiFi":{
"Hostname":"Wemos-Pool-Controller-7",
"IP config":"DHCP",
"IP":"10.0.0.XX",
"Subnet Mask":"255.255.255.0",
"Gateway IP":"10.0.X.X",
"MAC address":"5C:CF:7F:F0:28:8A",
"DNS 1":"10.0.X.X",
"DNS 2":"X.X.X.X",
"SSID":"XXXXXXXX",
"BSSID":"F0:9F:C2:A4:AE:EF",
"Channel":1,
"Connected msec":136965665,
"Last Disconnect Reason":201,
"Last Disconnect Reason str":"(201) No AP found",
"Number reconnects":26,
"RSSI":-53
},
"Sensors":[
{
"TaskNumber":1,
"Type":"Analog input - ADS1115",
"TaskName":"pH",
"TaskValues": [
{"ValueNumber":1,
"Name":"pH",
"Value":11.50}]
},
{
"TaskNumber":2,
"Type":"Analog input - internal",
"TaskName":"Pressure Filter",
"TaskValues": [
{"ValueNumber":1,
"Name":"Bar",
"Value":0.91}]
},
{
"TaskNumber":3,
"Type":"Display - LCD2004",
"TaskName":"Screen"},
{
"TaskNumber":4,
"Type":"Analog input - ADS1115",
"TaskName":"Orp",
"TaskValues": [
{"ValueNumber":1,
"Name":"Orp",
"Value":171.51}]
},
{
"TaskNumber":5,
"Type":"Switch input - Switch",
"TaskName":"PIR",
"TaskValues": [
{"ValueNumber":1,
"Name":"Switch",
"Value":0}]
},
{
"TaskNumber":6,
"Type":"Environment - DS18b20",
"TaskName":"Temperature",
"TaskValues": [
{"ValueNumber":1,
"Name":"Temperature",
"Value":9.06}]
},
{
"TaskNumber":7,
"Type":"Generic - Pulse counter",
"TaskName":"Water Flow",
"TaskValues": [
{"ValueNumber":1,
"Name":"Count",
"Value":0.00},
{"ValueNumber":2,
"Name":"Total",
"Value":0.00},
{"ValueNumber":3,
"Name":"Time",
"Value":0.00}]
},
{
"TaskNumber":9,
"Type":"Generic - Dummy Device",
"TaskName":"Test",
"TaskValues": [
{"ValueNumber":1,
"Name":"[Pressure Filter#Bar]+10",
"Value":0.00},
{"ValueNumber":2,
"Name":"Pressure Filter#Bar +10",
"Value":0.00},
{"ValueNumber":3,
"Name":"",
"Value":0.00},
{"ValueNumber":4,
"Name":"",
"Value":0.00}]
}
]
}

Output from debug:
Code: Select all
3 mai 2018 07:58:44
   GhostXML                        Debugging on.
   GhostXML Debug                  Debug level: 1

3 mai 2018 07:58:59
   GhostXML Debug                  Device has been updated. Blow state list up to Trigger and Control Page labels.
   GhostXML Debug                  validateDeviceConfigUi() method called.
   GhostXML Debug                  Device has been updated. Blow state list up to Trigger and Control Page labels.
   GhostXML Debug                  Stopping GhostXML device: ESP8266 Pool Level (json)
   GhostXML Debug                  Starting GhostXML device: ESP8266 Pool Level (json)
   GhostXML Debug                  Device has been updated. Blow state list up to Trigger and Control Page labels.
   GhostXML Debug                  Time since update (695) is greater than configured frequency (15)
   GhostXML Debug                  Found configured device: ESP8266 Pool Level (json)
   GhostXML Debug                  Refreshing device: ESP8266 Pool Level (json)
   GhostXML Debug                  *   Trying 10.0.0.56...
* TCP_NODELAY set
* Connected to 10.0.0.56 (10.0.0.56) port 80 (#0)
> GET /json HTTP/1.1
> Host: 10.0.0.56
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 242
< Connection: close
< Access-Control-Allow-Origin: *
<
{ [242 bytes data]
* Closing connection 0

   GhostXML Debug                  Source file type: JSON
   GhostXML Debug                  Device needs updating set to: True
   GhostXML Debug                  Pulling down existing state list.
   GhostXML Debug                    Writing dynamic states to device.
   GhostXML Debug                  Device needs updating set to: False

Posted on
Thu May 03, 2018 11:57 am
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

Took me a while, but I figured out what the issue is.

Some of the source json keys contain spaces (which I don't believe is good form) but when I updated the plugin to contend with the spaces, it then parses successfully. It's a wonder that this particular problem has taken this long to surface.

I'll post an update to the plugin later tonight (North America time). You should then be in business.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Thu May 03, 2018 6:16 pm
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

I just released a new version of the plugin to the Indigo Plugin Store that should be able to handle your JSON feed.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri May 04, 2018 3:35 am
qcvictor offline
Posts: 74
Joined: Apr 23, 2015
Location: Quebec, Canada

Re: New Plugin: GhostXML

Hi Dave
Thanks you so much for your great job and support, it's working now.
Vic

Posted on
Fri May 04, 2018 3:54 am
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

Excellent. Thanks for letting me know.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri Jun 08, 2018 1:25 pm
berkinet offline
User avatar
Posts: 3097
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Interesting problem

I tried the GhostXML plugin to read data from my RainMachine irrigation controller using their RESTful API. If I receive the JSON data from the RainMachine and save it into a file, the plugin reads the file and updates the pseudo device as expected -- very cool, thanks!. However, if I have the plugin connect directly to the RainMaching using the same URL that works in Safari
Code: Select all
https://rainmachine.local:8080/api/4/watering/past
Here is the log with debugging set to high
    GhostXML Debug Refresh frequency: 0 (Manual refresh only)
    GhostXML Debug refreshData() method called.
    GhostXML Debug refreshDataForDev() method called.
    GhostXML Debug Found configured device: new device
    GhostXML Debug Refreshing device: new device
    GhostXML Debug getTheData() method called.
    GhostXML Debug * Trying 192.168.5.45...
    * TCP_NODELAY set
    * Connected to 192.168.5.45 (192.168.5.45) port 8080 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
    * successfully set certificate verify locations:
    * CAfile: /etc/ssl/cert.pem
    CApath: none
    * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    } [512 bytes data]
    * TLSv1.2 (IN), TLS handshake, Server hello (2):
    { [89 bytes data]
    * TLSv1.2 (IN), TLS handshake, Certificate (11):
    { [957 bytes data]
    * TLSv1.2 (OUT), TLS alert, Server hello (2):
    } [2 bytes data]
    * SSL certificate problem: self signed certificate
    * stopped the pause stream!
    * Closing connection 0

    GhostXML Debug Source file type: JSON
    GhostXML Debug parseTheJSON() method called.
    GhostXML Error new device: No JSON object could be decoded
    GhostXML Debug cleanTheKeys() method called.
    GhostXML Error Error cleaning dictionary keys: 'NoneType' object has no attribute 'iterkeys'
    GhostXML Debug Refresh frequency: 0 (Manual refresh only)
And, here is the JSON data returned
Code: Select all
{"pastValues": [{"pid": 7, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 8, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 3, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.464572714257837, "qpf": 7.9099999999999993}, {"pid": 2, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 11, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 1, "dateTimestamp": 1528408800, "dateTime": "2018-06-08 00:00:00", "used": true, "et0": 13.567305490819034, "qpf": 3.1000000000000001}]}
The data read from the file is identical.

For completeness, here is what I get back from calling the same url in curl[, and which is then saved to file, minus the headers.
Code: Select all
HTTP/1.1 200 OK
Content-type: text/html
Content-Length: 881
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: *
X-Frame-Options: DENY
Date: Fri, 08 Jun 2018 08:14:18 GMT
Server: lighttpd/1.4.31

{"pastValues": [{"pid": 7, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 8, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.407131865431126, "qpf": 7.9099999999999993}, {"pid": 3, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.464572714257837, "qpf": 7.9099999999999993}, {"pid": 2, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 11, "dateTimestamp": 1528322400, "dateTime": "2018-06-07 00:00:00", "used": true, "et0": 12.201767677812171, "qpf": 8.1400000000000006}, {"pid": 1, "dateTimestamp": 1528408800, "dateTime": "2018-06-08 00:00:00", "used": true, "et0": 13.567305490819034, "qpf": 3.1000000000000001}]}+ exit 0

Posted on
Fri Jun 08, 2018 2:31 pm
DaveL17 offline
User avatar
Posts: 4195
Joined: Aug 20, 2013

Re: New Plugin: GhostXML

Thanks for reporting the error. I see the same one from time to time in my development environment on both URL-based as well as file-based sources. That said, the error isn't persistent and the device will right itself during a subsequent update. I believe the error is a result of code that isn't as thread-safe as it could be. I've been working on a fundamental refactoring which I hope will be considerably more stable and fully thread-safe that should address the issue. A couple of questions:

Does the error occur every time the device tries to update from the URL or only sometimes?
Do you have more than one GhostXML device configured?

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri Jun 08, 2018 2:48 pm
berkinet offline
User avatar
Posts: 3097
Joined: Nov 18, 2008
Location: Berkeley, CA, USA & Mougins, France

Re: New Plugin: GhostXML

DaveL17 wrote:
...Does the error occur every time the device tries to update from the URL or only sometimes? Do you have more than one GhostXML device configured?

Well, I let it run for a while with 15 sec updates and it never connected. I have 2 devices, But, had the file based device turned off while this was happening (Indigo device comms unchecked). FWIW, the file based device never failed.

I have one additional note. Initially I created the file based device, then I copied it and modified it to use a URL. Since I copied another device the state’s were already populated. That URL based device never worked, though it continued to display the states it had inherited . I then deleted it and created a new device from scratch, which continued to have the same connection problem but, only displayed the meta device states like last time connected.

Who is online

Users browsing this forum: No registered users and 1 guest