Graphite data logger

Posted on
Sat Dec 31, 2016 11:05 am
jh71283 offline
Posts: 127
Joined: Jun 16, 2014

Graphite data logger

Hi all

I have written a plugin to log various data to Graphite. Once the data is logged in Graphite, you can use something such as Grafana to produce some nice graphs and dashboards :


Screen Shot 2016-12-31 at 16.51.49.png
Screen Shot 2016-12-31 at 16.51.49.png (98.18 KiB) Viewed 7259 times


Screen Shot 2016-12-31 at 16.52.29.png
Screen Shot 2016-12-31 at 16.52.29.png (139.22 KiB) Viewed 7259 times



You can find the plugin at: https://github.com/jh71283/IndigoGreenSkyDataLogger

... To log a variable to Graphite, simply put it into a folder called Graphite. All variables in here will be sampled once a second and sent to Graphite. It's that simple!

Posted on
Sat Dec 31, 2016 3:16 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Graphite data logger

Thanks for sharing – that does produce some very nice looking graphs!

Image

Posted on
Sun Jan 01, 2017 6:16 am
AndyVirus offline
Posts: 257
Joined: Mar 18, 2014
Location: Newport Pagnell, UK

Re: Graphite data logger

Thanks for this!
I set up a graphite server VM using Synthesise (https://github.com/obfuscurity/synthesize) to try this out and works great.

2 things:

1) All variables in other folders do not get sent/logged to graphite (as you stated) but the ones that do not reside in a folder do, not sure if this is expected or a bug.
2) Is there a way to send ALL variables to Graphite as i already have lots of folders for Heating, blinds, movement etc and I don't want to duplicate these into a folder called graphite (with duplicate triggers) to get them into graphite.

Thanks again!

Posted on
Mon Jan 02, 2017 5:56 am
jh71283 offline
Posts: 127
Joined: Jun 16, 2014

Re: Graphite data logger

Hi Andy

No, I didn't expect that to be that case. The plugin specifically looks for the Graphite folder, and then loops through the variables in that folder so I am a bit stumped on that one!

I'll have a look at some other options for which variables get sent across too.

Posted on
Mon Jan 02, 2017 6:45 am
AndyVirus offline
Posts: 257
Joined: Mar 18, 2014
Location: Newport Pagnell, UK

Re: Graphite data logger

I only found that by accident as after a fresh install of both the plugin and graphite a few variables showed up unexpectedly. I dont have many that are not in folders, i think only a few pibeacon variables are not and i dint need to graph those. I can see in your plugin.py that the folder "graphite" is defined, if trivial is there a way to define additional folders? Thats all im after really so i can graph my "heating" folder etc.


Sent from my iPhone using Tapatalk

Posted on
Mon Jan 02, 2017 12:58 pm
jh71283 offline
Posts: 127
Joined: Jun 16, 2014

Re: Graphite data logger

OK I have just pushed version 2.0.1 to the github repo.

I've made it a major version change as it includes breaking changes.

When you upgrade to this version, you will need to tell the plugin which variables to log, otherwise it will not log any.

Variables now have their folder name appended to them prior to being sent to Graphite, meaning that any variables you already have, will change in Grafana.

indigo.Sheep and indigo.Dog will now become indigo.Graphite.Sheep and indigo.Graphite.Dog (Because they are from the Graphite folder)

However this of course means that there is no need to put your variables in the Graphite folder at all.

Posted on
Tue Jan 03, 2017 3:18 pm
blysik offline
Posts: 213
Joined: Jan 06, 2015

Re: Graphite data logger

This is very cool! However, if an Indigo variable or folder has a space in the name, you should replace that space with an underscore.

Posted on
Tue Jan 03, 2017 6:58 pm
blysik offline
Posts: 213
Joined: Jan 06, 2015

Re: Graphite data logger

Any chance this plugin could implement an Action? ie, "Log a data point to graphite", you supply the namespace, and either a variable, or a value, and it sends it one off to the configured graphite server.

Posted on
Wed Jan 04, 2017 1:15 am
jh71283 offline
Posts: 127
Joined: Jun 16, 2014

Re: Graphite data logger

Both good ideas, blysik!
I'll take a look.


Sent from my iPhone using Tapatalk

Posted on
Wed Jan 04, 2017 1:26 am
juntta offline
Posts: 143
Joined: Oct 13, 2014
Location: Finland

Re: Graphite data logger

Or how about utilizing the SQL logger database as data source?

That way both variables & device data could be used in Graphite.

Posted on
Wed Jan 04, 2017 2:27 am
jh71283 offline
Posts: 127
Joined: Jun 16, 2014

Re: Graphite data logger

My main problem with doing that is that a user then has to configure an extra system - postgres, and the SQL logger plugin.

And the other thing is that I find the schema of the SQL logger output a little odd, and incomplete.

A separate table per device / variable makes little sense to me, along with the fact that there is nowhere in that database that stores the names of the devices / variables.

I also can't remember if there is a master table containing a list of the devices / variables, or if you have to hunt the schema.

There opinions are of course, my own, and other people may love the way the database is laid out.

Posted on
Wed Jan 04, 2017 10:19 am
juntta offline
Posts: 143
Joined: Oct 13, 2014
Location: Finland

Re: Graphite data logger

I can understand your point - plugin that needs plugin which requires another plugin can be endless cycle!

Maybe some other plugin developers can give you hints how they've managed to tackle those database questions.

I think your approach is great but I wouldnt mind keeping same data in one place and avoid copying it.

Posted on
Sun Jan 15, 2017 3:06 pm
sparky21 offline
User avatar
Posts: 26
Joined: Nov 29, 2014
Location: Sierra Foothills, Ca

Re: Graphite data logger

This is cool. I live in graphite/grafana at my job, so its fun to see this. I'm motivated to setup an unused system to run graphite on now :-)

Posted on
Sat Jan 21, 2017 9:30 am
elov offline
Posts: 20
Joined: Dec 27, 2016

Re: Graphite data logger

Hi,

I want to share that I've made a fork of your plugin which includes this extended functionality:

Automatic enumeration of all indigo devices and their state values. So everything that can be casted into a float/int will be exported.
RegEx filters for both device names and state names, so I can remove or only include the devices and states I want.

So now I have an automated solution that send brightness, on/off states, curEnergyLevel and so forth for all my Z-wave devices automatically, simple set and forget without having to create any vars for each value. I even get RSSI value logged for every Wifi device connected through the Unifi plugin. :)

I'll get the code uploaded if anyone is interested with a small guide of how I setup InfluxDB and Grafana locally on my Mac mini running Indigo.

Cheers

Posted on
Sat Jan 21, 2017 12:47 pm
blysik offline
Posts: 213
Joined: Jan 06, 2015

Re: Graphite data logger

Sweet!


Sent from my iPhone using Tapatalk

Who is online

Users browsing this forum: No registered users and 13 guests