InfluxDB plugin

Posted on
Tue Jul 18, 2017 3:13 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

InfluxDB plugin

A little more involved to install, but useful for pretty graphs in Grafana. (Pretty graph below)

https://github.com/smurfless1/indigo-influxdb
Attachments
Screen Shot 2017-07-13 at 4.23.55 PM.png
Screen Shot 2017-07-13 at 4.23.55 PM.png (203.61 KiB) Viewed 6462 times

Posted on
Tue Jul 18, 2017 4:10 pm
jay (support) offline
Site Admin
User avatar
Posts: 18201
Joined: Mar 19, 2008
Location: Austin, Texas

Re: InfluxDB plugin

You're on a roll! Thanks for another cool plugin!

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Fri Jul 21, 2017 12:18 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

To backfill information from your logs so you have something to look at on the first day, you can use this:

https://github.com/smurfless1/indigo-backfill-influx

after the plugin is installed.

Posted on
Sat Jul 22, 2017 7:18 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

So this is a continuing conversation about problems seen in a different topic:
1. string vs. unicode errors, error in the log mentions something about encoding.
If you see this, please go to the configuration for the plugin and turn on the debug checkbox until you see one of those. Post an excerpt from your log in here for me, it helps.

2. This one is more general. I noticed early on that:
a. Influx doesn't do well when you store numbers as strings. It tries, but it falls down in many areas.
b. Many numbers get passed to me as strings.
c. I know how to change stringy-numbers to numbery-numbers.

AWESOME. So if I can, I do. Suddenly, my charts start to work. This was version 0.5.0.

Now. The first time it goes into InfluxDB, Influx will say:
HEAR YE HEAR YE: THIS COLUMN IS FOREVERMORE A NUMBER

Now let's say that a few minutes later, something comes in in the same column, but it's totally a string, like "Couldn't get this value". I don't really have a good way to know "the last time I posted this column it was type number, so I should make a different decision" much less what decision to make. Similarly, I can't always tell what type a state variable is supposed to be, but I know for sure I can't plot anything on a line graph that has "Couldn't get this value" instead of 72.5. So what you get currently is Influx telling you, "Hey, when I made this, it was a float, but now you're passing me a string. WHAT THE HECK DUDE?!"

So now comes the audience participation portion of the project. I need debug output (in the plugin configuration, checkbox on) between one that works and one that doesn't for devices I don't have that do this. In the meantime I'll see what I can figure out for variables I see that are like this.

Posted on
Sat Jul 22, 2017 10:52 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

So I found a solution, but you'll have to delete the database you have so far. I know! It sucks. Sorry!

To do that:
Get the latest (0.5.1) version from the zipfile on github.
Install and then open the configuration.
Click on the "Erase all my data" checkbox
Then reload the plugin.
Then remember to uncheck it after it's connected. It only happens when the plugin is loaded.

What I'm doing now is creating a -separate- numeric field if I can, but leaving the original alone. Meaning for a very few things, you'll get things like this:
thing.temperatureF = "95"
thing.temperatureF.num = 95

In your graphs, you'll have better success with *.num, and now if it doesn't convert to a number, the value simply won't be there to get in the way. There's no promise that the devices and plugins you use will still try this trick with the raw field, but that's outside my control, and at least this way I won't be causing trouble, just trying to make something smarter by duplicating the ones I can use.

... and now I have to wait until morning so I get some hits from some of my modules to graph. Night!

Posted on
Sun Jul 23, 2017 10:30 pm
sbe08 offline
Posts: 30
Joined: Sep 14, 2006

Re: InfluxDB plugin

0.5.1 has corrected all errors that I experiencing. No more issues with devices created by plug-ins or Unicode encoding errors. Good work!

I tried to overload the necessary functions to add variables into Influx, but it didn't work on my first attempt. Shouldn't be very complicated given that you did all the heavy lifting already. Here's what I tried:

Code: Select all
    def variableUpdated(self, origDev, newDev):
        # call base implementation
        indigo.PluginBase.variableUpdated(self, origDev, newDev)

        # custom add to influx work
        # tag by folder if present
        tagnames = 'name folderId'.split()
        newjson = self.adaptor.diff_to_json(newDev)

        newtags = {}
        for tag in tagnames:
            newtags[tag] = unicode(getattr(newDev, tag))

        # add a folder name tag
        if hasattr(newDev, 'folderId') and newDev.folderId != 0:
            newtags['folder'] = indigo.variables.folders[newDev.folderId].name

        if self.pluginPrefs.get('debug', False):
            indigo.server.log(json.dumps(newjson).encode('utf-8'))

        self.sendVariable(newtags, newjson)

    def sendVariable(self, tags, what):
        json_body=[
            {
                'measurement': 'variable_changes',
                'tags' : tags,
                'fields':  what
            }
        ]
        try:
            self.connection.write_points(json_body)
        except Exception as e:
            indigo.server.log("InfluxDB write error:")
            indigo.server.log(unicode(e))


Posted on
Mon Jul 24, 2017 10:24 am
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

Grab 0.5.2:
Publish changes to variables in the measurement variable_changes
Publish changes to NOAA Weather stations to measurement weather_changes
Publish changes to thermostat-class devices to measurement thermostat_changes

db

Posted on
Mon Jul 24, 2017 1:58 pm
neilk offline
Posts: 714
Joined: Jul 13, 2015
Location: Reading, UK

Re: InfluxDB plugin

Hi,
I have PiBeacon running and I am getting the following errors

Code: Select all
InfluxDB Plugin                 InfluxDB write error:
   InfluxDB Plugin                 400: write failed: field type conflict: input field "state.Pi_4_Distance" on measurement "device_changes" is type int64, already exists as type float

   InfluxDB Plugin                 InfluxDB write error:
   InfluxDB Plugin                 400: write failed: field type conflict: input field "state.TxPowerSet" on measurement "device_changes" is type float64, already exists as type integer


and
Code: Select all
InfluxDB Plugin                 InfluxDB write error:
   InfluxDB Plugin                 400: write failed: field type conflict: input field "state.Pi_7_Distance" on measurement "device_changes" is type int64, already exists as type float

   InfluxDB Plugin                 InfluxDB write error:
   InfluxDB Plugin                 400: write failed: field type conflict: input field "state.TxPowerSet" on measurement "device_changes" is type float64, already exists as type integer

   InfluxDB Plugin                 InfluxDB write error:
   InfluxDB Plugin                 400: write failed: field type conflict: input field "state.PosX" on measurement "device_changes" is type float64, already exists as type integer

   piBeacon                        filling empty state lastStatusChange  of dev:beacon-72:47:3A:E2:8C:43   with current date: 2017-07-24 20:54:30
   InfluxDB Plugin                 InfluxDB write error:
   InfluxDB Plugin                 400: write failed: field type conflict: input field "state.PosX" on measurement "device_changes" is type float64, already exists as type integer



I have got as far as getting grafana up and data is getting through, but I am still fumbling to get my first dashboard running. Enjoying the journey and getting this running was fun as various earlier uneducated playing with MacPorts and HomeBrew had things in a mess. Also in the readme, is their a typo in the alias command, I needed to move the quotes until after the reference to the database name then all was good.

Thanks,
Neil

Posted on
Mon Jul 24, 2017 8:19 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

I'm guessing that Influx is having trouble guessing the format of the field the first time it goes in. I downloaded the plugin, let me have a look inside.

Posted on
Mon Jul 24, 2017 10:10 pm
sbe08 offline
Posts: 30
Joined: Sep 14, 2006

Re: InfluxDB plugin

Version 0.52 stores all the variable states in InfluxDB, but I'm unable to build a working dashboard. It seems that you're not doing the same magic trick with variables as you did with devices. The value is stored as text when it could be interpreted as a float or int. Perhaps a _num column would make it perfect. Then, I can stop annoying you!

In my Indigo setup, almost everything happens through variables... For instance, each motion detector has a counter variable that increment every minute of inactivity. This is used to trigger everything else, from lighting to audio setup... Having the ability to drill down and compare variable data with device states and environmental sensors will allow me to pinpoint improvement in my system. I have 310 devices, 280 triggers, 220 action groups and 130 scheduled events, so there's a lot happening in parallel! :-)

Posted on
Tue Jul 25, 2017 10:32 am
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

So I just realized, they gave me everything I need to cast it to the right type, so I put in a loop that goes through and every time it complains, I change it to that type, and send it back. So far, it's working, but give me a day or so to work out some kinks in case it does something odd. Like I'm not sure if it's duplicating data, and that would be annoying.

Rules: it only retries up to 30 times, and if it can't actually change it from say "no number here" to a number, it will just go on with the partial record already written and leave things be.

Appending: Influx (the project) is at least mentally ready to give up on integers, so I nearly everything is being coerced to a float by default (if your database is empty). But the new type-retry loop will notice if you don't change it over and only complain the first time around.

I also updated the influx backfill script with this loop, and it will try to upload debug blocks from your logfiles if you had debug on. So more data makes it in, and debug blocks, while ugly to look at, put your data in a second place while we work out the kinks.

Posted on
Thu Jul 27, 2017 3:30 am
sbe08 offline
Posts: 30
Joined: Sep 14, 2006

Re: InfluxDB plugin

Good to hear you're making progress! As always with Indigo projects, there's no rush and it's only for fun! (At least for me). I'm still storing everything in PostgreSQL, so it's not a problem to delete the Influx database and start over if necessary. Thanks again for your involvement!

Posted on
Sun Jul 30, 2017 7:55 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

Version 0.5.3:
* Retries posting to Influx if Influx complains about a type. As described, seems to be working great.
* Backfill is more aware of these changes and will try to publish the same metrics
* The Giants are STILL losing, so I have lots of time to figure this kind of nonsense out. Please make it stop.

Posted on
Fri Aug 04, 2017 12:38 am
koensayr offline
Posts: 90
Joined: Jul 10, 2013

Re: InfluxDB plugin

Do you have any guides or available templates to create the grafana dashboards that you show in your first post? Specifically I'm interested in the bar chart you made for which lights were on at which times.

-Josh

Posted on
Mon Aug 14, 2017 10:18 pm
sbe08 offline
Posts: 30
Joined: Sep 14, 2006

Re: InfluxDB plugin

If you look in the plug-in, you'll find the .json code for his dashboards. You'll need to install Discreet module for Grafana to make them work.

Who is online

Users browsing this forum: No registered users and 1 guest