Valid JSON syntax

Posted on
Sun Dec 12, 2021 9:51 am
johnpolasek offline
Posts: 897
Joined: Aug 05, 2011
Location: Aggieland, Texas

Valid JSON syntax

Hate to display my ignorance, but what is a valid JSON syntax... My old sites age power monitor died that I could poll via XML, and I replaced it with an Emporia unit (cloud based but a lot cheaper than a new SiteSage)... The emporia only has the ability to post a JSON file and I had hoped to simply change the status and then pick out the data as strings and on convert them via python to variables I could use. The JSON file had the form:


    {"channel":"1,2,3","deviceGid":"27843","usageList":[null,null],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"1","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[null,null]}
    {"channel":"2","deviceGid":"27843","usageList":[null,null],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"3","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[null,null]}
    {"channel":"4","deviceGid":"27843","usageList":[null,null],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"5","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[null,null]}
    {"channel":"6","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[null,null]}
    {"channel":"7","deviceGid":"27843","usageList":[null,null],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"8","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[null,null]}
    {"channel":"9","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[null,null]}
    {"channel":"10","deviceGid":"27843","usageList":[8.424741331736246E-4],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"11","deviceGid":"27843","usageList":[0.0016715458806355794],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"12","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[1.4420307079950968E-5]}
    {"channel":"13","deviceGid":"27843","usageList":[0.0],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"14","deviceGid":"27843","firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[0.008018433159722222]}
    {"channel":"15","deviceGid":"27843","usageList":[8.562559096018473E-4],"firstUsageInstant":"2021-12-04T22:30:00Z"}
    {"channel":"16","deviceGid":"27843","usageList":[0.0],"firstUsageInstant":"2021-12-04T22:30:00Z"}
However, the Ghost debug reports
    GhostXML Debug self.managedDevices: {}
    GhostXML Debug [sitesage] Communication started.
    GhostXML Debug [sitesage] Parse Error: Extra data: line 2 column 1 - line 18 column 1 (char 107 - 1807)
    GhostXML Debug [sitesage] jsonRawData
    GhostXML Warning There was a parse error. Will continue to poll.
    GhostXML Debug self.managedDevices: {643899493: <__main__.PluginDevice object at 0x10e702190>}
    GhostXML Debug [sitesage] Communication stopped.

Are there any options I can use in GhostXML to parse this into devices, or will I have to write my own upfront parser to convert it to an XML?

Posted on
Sun Dec 12, 2021 11:11 am
DaveL17 offline
User avatar
Posts: 6038
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Valid JSON syntax

Is that the full return from the device? That's not a valid JSON format (but I suppose would be a bunch of small JSON payloads). There are JSON validators online that allow you to copy and paste code to test it. This is a valid sample from http://json.org.

Code: Select all
{
    "glossary": {
        "title": "example glossary",
      "GlossDiv": {
            "title": "S",
         "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
               "SortAs": "SGML",
               "GlossTerm": "Standard Generalized Markup Language",
               "Acronym": "SGML",
               "Abbrev": "ISO 8879:1986",
               "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                  "GlossSeeAlso": ["GML", "XML"]
                    },
               "GlossSee": "markup"
                }
            }
        }
    }
}

JSON is like a Python dictionary (or vice versa) in the way they appear, but there are differences. A JSON payload is a string where a dictionary is a Python object. The GhostXML plugin attempts to convert valid JSON into a dictionary object to do its work. It's not a one-to-one conversion because of subtle differences between the two. There are also some differences in how Indigo works with dictionaries (allowable keys, for example) and the plugin attempts to make the necessary changes. We've found that some "illegal" characters can crop up from time to time (we haven't seen that happen for quite a while so it feels like we have a decent handle on that).

I wouldn't recommend converting the JSON to XML as there are considerable differences between the two. For a small payload it's doable, but I'd suggest finding a way to work with the JSON you have if possible.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun Dec 12, 2021 11:36 am
FlyingDiver offline
User avatar
Posts: 5599
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Valid JSON syntax

If you're going to be writing a Python script to parse the JSON, then no point in going to XML. Just populate the variables directly in the script.

That does look like a sequence of JSON strings. Reading those from a file line by line and parsing the JSON is easy. Depending on how you're getting the file, you might be able to just read them from the source and not use a file at all.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Dec 12, 2021 12:40 pm
johnpolasek offline
Posts: 897
Joined: Aug 05, 2011
Location: Aggieland, Texas

Re: Valid JSON syntax

As I said, I don't know JSON syntax; looking at your example, it would seem that it's the embedded ":" without enclosing them in "{", meaning that I would have to add something like:
Code: Select all
{Channels:
   {"channel 1,2,3":
      {"deviceGid":"27843","usageList":[null,null]","firstUsageInstant":"2021-12-04T22:30:00Z"
      }
   }
   {"channel 1":
      {"deviceGid":"27843,"firstUsageInstant":"2021-12-04T22:30:00Z","usageList":[null,null]
      }
   }
   .
   .
   .
   {"channel 16":
      {"deviceGid":"27843","usageList":[0.0],"firstUsageInstant":"2021-12-04T22:30:00Z"
      }
   }
}


in effect adding the Enclosing {Channels: and consolidating the "Channel" and channel # to make each item unique, with all the rest of the line being converted to states (DeviceID, usageList, and firstUsageInstant) within each Channel # item, possibly replacing the [null,null] items with 0.0 on the fly , since I have no idea why empoeria returns the Channel [1,2,3] at the top and [null][null] for some power rings (like the hot water heater) that have no load , but 0.0 for others in the "usageList" item.

And I am using a third party java utility from GitHub (https://github.com/helgew/emporia-downloader) since the company has stated on the user forums that they have no plans EVER to have an API beyond their website that allows a manual CSV download from their website... which numerous of us have complained about.;

Posted on
Sun Dec 12, 2021 2:02 pm
DaveL17 offline
User avatar
Posts: 6038
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Valid JSON syntax

Do you have a URL where you can download the CSV from the website directly? That would make your life a whole lot simpler.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue Dec 14, 2021 5:07 pm
johnpolasek offline
Posts: 897
Joined: Aug 05, 2011
Location: Aggieland, Texas

Re: Valid JSON syntax

Unfortunately, nothing that I can call programmatically; all I can do is go to their website and manually enter a time and date to start and and end and have it send me a link to download the data... a DIFFERENT link every time I make a request. I don't get these guys; it's like they are doing everything they can think of to KEEP people from using their product... no local access, no easy download, the public domain JSON grabber is apparently using undocumented features.

Posted on
Tue Dec 14, 2021 5:33 pm
DaveL17 offline
User avatar
Posts: 6038
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Valid JSON syntax

That's too bad. We'll have to work with what we've got. Let me play around with the data above and get back to you (I have one commitment to finish first, but it shouldn't take too long).

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue Dec 14, 2021 5:48 pm
FlyingDiver offline
User avatar
Posts: 5599
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Valid JSON syntax

You should see if your browser keeps a history of the URLs fetched. You might be able to recreate what the browser is doing using Requests in Python. Then you could automate the process.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Tue Dec 14, 2021 7:59 pm
DaveL17 offline
User avatar
Posts: 6038
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Valid JSON syntax

This is one approach, just as a starting point. It presumes the source data are in a file called "emporia.txt". It doesn't have to be done this way--there's lot's of possibilities.

Code: Select all
from ast import literal_eval

# Read data from file
with open('emporia.txt', 'r') as infile:
   lines = infile.readlines()

# Convert the string representation of the data to individual dicts and get rid of any "null"s.
new_lines = [literal_eval(line.replace('null', 'None')) for line in lines]

# Combine into a single dict (optional)
line_dict = {}
x = 1
for line in new_lines:
   line_dict[x] = line
   x += 1
   
# From here you can assign values to variables (or whatever)
print(line_dict)
Result:

Code: Select all
{1: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '1,2,3', 'deviceGid': '27843'}, 2: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '1', 'deviceGid': '27843'}, 3: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '2', 'deviceGid': '27843'}, 4: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '3', 'deviceGid': '27843'}, 5: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '4', 'deviceGid': '27843'}, 6: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '5', 'deviceGid': '27843'}, 7: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '6', 'deviceGid': '27843'}, 8: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '7', 'deviceGid': '27843'}, 9: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '8', 'deviceGid': '27843'}, 10: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [None, None], 'channel': '9', 'deviceGid': '27843'}, 11: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [0.0008424741331736246], 'channel': '10', 'deviceGid': '27843'}, 12: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [0.0016715458806355794], 'channel': '11', 'deviceGid': '27843'}, 13: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [1.4420307079950968e-05], 'channel': '12', 'deviceGid': '27843'}, 14: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [0.0], 'channel': '13', 'deviceGid': '27843'}, 15: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [0.008018433159722222], 'channel': '14', 'deviceGid': '27843'}, 16: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [0.0008562559096018473], 'channel': '15', 'deviceGid': '27843'}, 17: {'firstUsageInstant': '2021-12-04T22:30:00Z', 'usageList': [0.0], 'channel': '16', 'deviceGid': '27843'}}

From here you can assign data pretty easily.
Attachments
emporia.txt
(1.76 KiB) Downloaded 12 times

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Dec 18, 2021 3:42 pm
johnpolasek offline
Posts: 897
Joined: Aug 05, 2011
Location: Aggieland, Texas

Re: Valid JSON syntax

Thanks; mental blindness... If I'm going to have to tear the input file apart anyway, I mighty as well go straight to variables without going through Ghost at all... I just wish the jerks at Emporia wouldn't have made it so tough to get the data... and hope they won't close the port that I am using to pull their data off the website. You have to wonder why they don't want to unload their servers and give a wider base of customers the data.

Posted on
Sat Dec 18, 2021 4:11 pm
DaveL17 offline
User avatar
Posts: 6038
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Valid JSON syntax

There are several routes you could take. Another would be to convert the data to JSON, save that to a file, and then load the file into GhostXML. All you would be gaining by that approach would be to have the data converted to device states. However, if you're just using the data for things like control pages and firing triggers and such, putting the data into variables would also work well.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 1 guest