Integration with InfluxDB and Grafana for graphing

Posted on
Wed Jul 12, 2017 11:03 am
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

I think I'd like to get this written in python (as a plugin?), as it just seems like a logical step, and I'd love to tinker with it. And I'm handy enough in python. Let me see if I can figure out the best integration path. Hints appreciated if anyone has them!

No promise on schedule, but I'll get started.

Posted on
Wed Jul 12, 2017 2:47 pm
jtodd offline
Posts: 75
Joined: Apr 15, 2014

Re: Integration with InfluxDB and Grafana for graphing

That would be great! Especially if you can figure out how to import all device data. Right now, my script fails at collecting a bunch of the values that are specific about devices, because the API doesn't expose them. I've used a hack where within Indigo I read a device value and insert it into a variable, and since I can read variables that's how I get the data export-able into InfluxDB. This seems un-necessary and somewhat silly, and if a plugin can get better access that would be great. I haven't touched this stuff in a year or two but it just keeps on running...

JT

Posted on
Wed Jul 12, 2017 3:34 pm
jay (support) offline
Site Admin
User avatar
Posts: 16821
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Integration with InfluxDB and Grafana for graphing

If you just need to write a formatted file for import into InfluxDB, that should be quite simple for a Python script, it's very good at that sort of thing.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Jul 13, 2017 9:25 am
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

"Early beta"

perfectly functional, but I sure don't know what I'm doing with Grafana and InfluxDB yet. The data is definitely in there, but I have a little more to do still, like create a home page for it, make sure all the interesting fields are present, etc.

In short: I have both a InfluxDB plugin and Kafka Producer plugin that both write json for every device change. They both write every state that is sent from Indigo, as well as the name and timestamp. I have to figure out how to write a couple of the more structured fields in a way that would be interesting to users.

Installation is not super-pretty, but at least it's scripted, as both of these depend on externally installed python modules. As a homebrew user, I had to figure out how to correctly install them where Indigo can get them - thus the script.

By the time I went to bed I was having Grafana graph the thermostat temperatures vs. the local airport temperatures, all from data Indigo sent me. There are a couple things that I don't know how to do, but might not be strictly plugin material. Still tinkering - back later.

Plan B was to write the pure python Indigo plugin to pipe the changes out to an external python program, and have an entirely external python program that could load modules from homebrew/etc. that ONLY publishes to the server, but that just doesn't seem necessary today. Tonight I might disagree, because that might be easier than explaining to users how to install those modules over and over.
Attachments
Screen Shot 2017-07-13 at 10.13.39 AM.png
yep, it's hot outside
Screen Shot 2017-07-13 at 10.13.39 AM.png (110.43 KiB) Viewed 2520 times

Posted on
Thu Jul 13, 2017 9:36 am
jtodd offline
Posts: 75
Joined: Apr 15, 2014

Re: Integration with InfluxDB and Grafana for graphing

Looks great - looking forward to seeing how you did it.

Posted on
Thu Jul 13, 2017 3:03 pm
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

I need feedback from Influx experts, which I am not. Install this version and tell me what I need to add. I think I need to figure out what tags I should be adding. Point me in the right direction and we can keep adjusting.

Grab the zip from below. Follow the "Read me and How to" to get it working, I tried to make it as straightforward as possible. Follow or copy-paste the instructions to a terminal. As it does add pip and two modules to the system python, I want you to understand that before it gets done.

More puttering as time continues.

https://github.com/smurfless1/indigo-influxdb

Posted on
Sun Jul 16, 2017 2:49 pm
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

Oooh, found a big bug - everything was being posted as a string so you can't do averages/sums/etc. But lightning hit our block yesterday and I'm rewiring things - fix coming ASAP.

Posted on
Tue Jul 18, 2017 3:02 pm
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

Grab a fresh zip, bug fixed.
Also a new toy: the Ping sensor. Pings a target IP and sets an on/off state. Then you just set a trigger to reset your modem when something goes down.

The kafka plugin will get an update when I have time to refactor it to use the same json class object.

Posted on
Tue Jul 18, 2017 3:08 pm
jtodd offline
Posts: 75
Joined: Apr 15, 2014

Re: Integration with InfluxDB and Grafana for graphing

Will take look shortly when I get some free time...

Does your module happen to go back into the Indigo database and pull out past data and stuff it into InfluxDB? That would be a big encouragement for people to try it out (and have some useful data) while experimenting with Grafana or whatever they're putting in to digest data out of Influx.

JT

Posted on
Thu Jul 20, 2017 4:29 pm
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

History: not yet, it subscribes to device changes. I have another python toy for reading device on/off events from logs, which is what eventually led me here. It doesn't find as many things, which is a bummer, because those temperature graphs are GREAT.

I think I could whip something up, but as a script instead of a plugin. You'd at least get -something- to look at.

Also the Ping, Kafka, and Influx modules are now separate downloads.

Posted on
Fri Jul 21, 2017 8:53 am
sbe08 offline
Posts: 29
Joined: Sep 14, 2006

Re: Integration with InfluxDB and Grafana for graphing

I've been able to set up InfluxDB and Grafana as per your instructions and a bit of tinkering... However, I'm getting a few errors that are probably due to text encoding:

Code: Select all
   InfluxDB Plugin Error           Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 97, in deviceUpdated
  File "/Library/Application Support/Perceptive Automation/Indigo 7/Plugins/InfluxDB.indigoPlugin/Contents/Server Plugin/indigo_adaptor.py", line 117, in diff_to_json
  File "/Library/Application Support/Perceptive Automation/Indigo 7/Plugins/InfluxDB.indigoPlugin/Contents/Server Plugin/indigo_adaptor.py", line 90, in to_json
  File "/Library/Application Support/Perceptive Automation/Indigo 7/Plugins/InfluxDB.indigoPlugin/Contents/Server Plugin/indigo_adaptor.py", line 56, in smart_value
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 618: ordinal not in range(128)


I never had to play with Unicode, but I'm guessing the the str() function to convert text is not the best practice...

Posted on
Fri Jul 21, 2017 9:53 am
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

Ah, just I have boring device names and state variables. In an effort to cure this boringness, I've renamed some of my closets.

My wife keeps turning on Monsioür Fabio and Pepé Le Pew. Rowr.

Just the same, this now looks fixed. Grab a fresh zipfile.

I also whipped up a script to read from Indigo 7 logs and write as much as I have figured out to the same database, so there's at least something in some of my graphs.
https://github.com/smurfless1/indigo-backfill-influx

Posted on
Fri Jul 21, 2017 9:22 pm
sbe08 offline
Posts: 29
Joined: Sep 14, 2006

Re: Integration with InfluxDB and Grafana for graphing

:D Not only you have introduced some excitement in your wife's life, but you just solved the error I had this morning! BTW, I've been looking for a way to add dashboards to Indigo for a long time, never finding the right solution. I had to pick you my jaw from the floor this morning when I saw the announcement on Indigo's Facebook page. What you did is just awesome!

I'm getting my database populated, influx now have 6000 entries from today's events. I'm a bit scared to try to your backfill script. Might try to see if I can migrate the data directly from my ProgresSQL database.

Do you also get some errors like this? They come from updates on the external devices defined by plug-ins. I get them from MQTT and Lutron devices.
Code: Select all
   InfluxDB Plugin                 InfluxDB write error:
   InfluxDB Plugin                 400: {"error":"partial write: field type conflict: input field \"state.topicMessage\" on measurement \"device_changes\" is type float, already exists as type string dropped=1"}


Oups, InfluxDB just spitted out another ASCII/Unicode error:

Code: Select all
   InfluxDB Plugin Error           Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 101, in deviceUpdated
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 20: ordinal not in range(128)

   InfluxDB Plugin Error           Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 101, in deviceUpdated
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 20: ordinal not in range(128)

Posted on
Sat Jul 22, 2017 7:48 am
Different Computers offline
User avatar
Posts: 2124
Joined: Jan 02, 2016
Location: East Coast

Re: Integration with InfluxDB and Grafana for graphing

I've been looking for a way to add dashboards to Indigo for a long time, never finding the right solution. I had to pick you my jaw from the floor this morning when I saw the announcement on Indigo's Facebook page. What you did is just awesome!


I'm missing the implication here, or perhaps the implementation. I know what a dashboard is in concept--in what way does this new functionality enable something important to that that was missing? I'm currently using MatPlotLib for graphing some Indigo stuff, and I do see that this InfluxDB/Grafana can do more than that, but I'm not seeing it doing so much more that it makes a dashboard display suddenly possible where it wasn't before.

SmartThings refugee, so happy to be on Indigo. 10.13.6 on an i5 MBP w/Harmony Hub, Hue, DomoPad, Dynamic URL, Device Extensions, HomeKit Bridge, MatplotLib, Plex, uniFAP, Wunderground, Nanoleaf, LED Simple Effects, Airfoil Pro, Grafana.

Posted on
Sat Jul 22, 2017 11:27 pm
smurfless1 offline
User avatar
Posts: 35
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: Integration with InfluxDB and Grafana for graphing

I'm afraid we've hijacked this thread a bit, let's move this discussion over here:
viewtopic.php?f=134&t=18676

By all means join me over there! And thanks for the hospitality.

Who is online

Users browsing this forum: No registered users and 2 guests

cron