So I found Karl Wachs' IndigoPlotD (viewtopic.php?f=165&t=11983) which is AMAZINGLY useful. I quickly created lots of different graphs, and his tool is super-flexible for getting things done. However... the interface is pretty awful, and I found that I was spending HOURS (actually, days) trying to frenzy-click my way through all of the pages of stuff and finding subtle bugs. This isn't his fault (I think) - it's a limitation of the way Indigo treats plugins. Having to add all the values I wanted to graph, and then defining what kind of values they were, and then defining the graphs, and then defining each line - it was super-tedious, and I found several bugs that caused me to have to start from scratch a few times.
I wrote a script to pull things from my SMA Solar (Sunnyboy) panels via SMASpot, and that kind of broke the bank. https://www.indigodomo.com/library/369/ I had many variables that I wanted to track, but every menu now had dozens and dozens of options which made working with IndigoPlotD extremely difficult.
So I wanted something that would "just work" with Indigo, and store and plot all of my values for EVERYTHING. Disk is cheap, CPUs are fast - just log it all and sort it out later. Personally, I like the GNUPlot look better from IndigoPlotD but I just burned myself out trying to get it working with all the stuff I had.
There are a bunch of new-ish graphing tools that are in use by the DevOps community, and it seemed that the data in Indigo was a good match for some of those tools. I've looked at InfluxDB with Grafana on top of it, which seemed to work well for operational tasks There are many other solutions (graphite, postgres, OpenTSDB, etc.) for graphing and time series data but I picked InfluxDB and Grafana because it seemed to be the quickest route for a RESTful import of Indigo data.
So I wrote this script to pull everything out of the REST APIs on Indigo and stuff them into InfluxDB.
There is a shortcoming to this script, and that is that not all possible values from Devices are plotted. See viewtopic.php?f=109&t=14358#p105051 for a conversation on this problem, but it's not major and there is a workaround of setting variables to the device data that you really want.
I am running it (currently) on a demo instance hosted at InfluxDB.com but I'll migrate to a VM running InfluxDB and Grafana located at home shortly. Note that InfluxDB and Grafana are NOT proprietary services, but are program suites that are open-source - I just chose to get started on an instance that was a freebie running as a cloud instance. Converting to your own instance should be super-trivial - just change the hostname/port/authentication data. Make sure you've created a database, then give the username you have created the correct permissions (i.e.: 'grant ALL on "indigo-data" to indigowrite')
Once I had the data flowing out, it only took me about 10 seconds to create the graph I show as an attachment. Adding lines to graphs, or whole graphs to dashboards is incredibly easy, and there are enough options to make it quite useful.
One of the things I couldn't figure out was the timestamps in the API data. They were kind of wacky, and didn't really look like UNIX Epoch timestamps - they were sometime in the 1980's if I converted them. I even tried subtracting them from currenttime, which led to dates around Jan 1, 2000. They were too long to be seconds since change - they were 500,000,000 or more in many cases and I've rebooted/restarted/updated devices much more recently than that. So I just left timestamps out of the data sent to InfluxDB, since it uses "now" as the default time stamp which is good enough accuracy for current data.
If someone would want to write an importer that takes historic SQL Logger data from Indigo and pushes it to InfluxDB, that would be amazingly useful to add here. Also this whole thing should be re-written in Python, but I don't know Python well enough so this hack will have to do.
- Code: Select all
#!/bin/bash
#
# (c) 2015 John Todd jtodd@loligo.com
#
# Indigo to InfluxDB API converter
# v0.1 2015/11/06
#
# This script connects to the Indigo (www.indigodomo.com) home automation
# system and scrapes all the device and variable information and deposits
# it into an InfluxDB using the RESTful APIs of both systems.
#
# You can use your own InfluxDB system, or something like https://customers.influxdb.com/
#
# Once the data is in InfluxDB, you can graph with tools like Grafana
# which will automatically look through InfluxDB tables and have a very
# nice interface.
#
# This script would need to be set up to execute in a crontab of some sort,
# so that it runs every minute. Something like this:
# * * * * /Users/johndoe/indigoinflux.sh
#
# Lots of things are not well-handled here. For one, many of the values
# in devices won't be captured since they're not visible via the API.
# You might have to create variables within Indigo and set the variables
# with the "hidden" device values so that they can be imported.
#
# This imports ALL device values and variables. If you have a very large
# number, this might be problematic. However, InfluxDB is pretty good at
# handling lots of variables and disk is cheap these days.
#
# If you have device values or variables that are strings, they won't be
# pushed into InfluxDB. If you wish, there are sections in both the device
# and variable section that let you convert an alphanumeric into a number for
# well-known values like "Off" or "activated".
#
# Note: this is an awful hack. I really am not a programmer, and this is bash
# scripting at its most brutal and inelegant. It was written in about two hours
# after giving up in frustration with other tools, so there has been little
# checking or serious thought given to streamlining.
# Please, for the love of all things good and holy, re-write this into something
# better and re-submit.
#
# What would really be useful would be a script that pulls all the data out
# of the SQL logger and pushes it into InfluxDB with timestamps included...
#
#
# Indigo data below here
user=demo
pass=myindigopasswordhere
site="127.0.0.1"
port="8176"
# InfluxDB data below here
iuser=indigowrite
ipass=blahblahDEblah
idb=indigo-data
isite="the-hostname-I-got-from-influxdb.c.influxdb.com"
iport="8086"
# Misc below here
wgetbin=/usr/local/bin/wget
wgethost="http://$site:$port"
wgetuser="--user=$user --password=$pass "
wgetcmd1="$wgetbin -q $wgetuser $wgethost"
tmpdir=/tmp/$site-$port
curl=/usr/bin/curl
oldIFS=$IFS
# housekeeping
# make sure the directories exist, delete the old cruft
rm -R $tmpdir
mkdir $tmpdir
mkdir $tmpdir/devices
mkdir $tmpdir/variables
# Start with fetching the devices from Indigo.
#
function getdevices {
IFS=$oldIFS
# get the devices file
cd $tmpdir;$wgetcmd1/devices.txt
# first, fetch all the device values from the Indigo server, ensuring
# that no stupid spaces screw things up
IFS=$'\n'
for device in `cat $tmpdir/devices.txt`
do
devicename=`echo $device | cut -f 3 -d "/"`
IFS=$oldIFS
cd $tmpdir/devices;`$wgetbin -q $wgetuser "$wgethost$device"`
done
# Next iterate through all those files and search for stuff
#
IFS=$'\n'
for device in `ls $tmpdir/devices`
do
rawstate=`grep displayRawState $tmpdir/devices/$device|tr -d " "|cut -f 2 -d :`
lastchanged=`grep "lastChanged :" $tmpdir/devices/$device|tr -d " "|cut -f 2 -d :`
if ! [ -z "$rawstate" ];
then
# clear up weird characters, spaces in measurement name
cleandevice=${device//[^a-zA-Z0-9_-]/}
cleandevice=`echo $cleandevice | rev | cut -c 4- | rev`
# now, clear up values to be float.
# If you have special strings that you want to match and
# set to 0 or 1 or whatever, this is the place to do it.
# Note that booleans are not supported. Feel free to patch.
if [ $rawstate == "off" ]; then rawstate="0"; fi
if [ $rawstate == "inactive" ]; then rawstate="0"; fi
if [ $rawstate == "disconnected" ]; then rawstate="0"; fi
if [ $rawstate == "unavailable" ]; then rawstate="0"; fi
if [ $rawstate == "active" ]; then rawstate="0"; fi
if [ $rawstate == "on" ]; then rawstate="0"; fi
if [ $rawstate == "connected" ]; then rawstate="0"; fi
if [ $rawstate == "Receiving" ]; then rawstate="0"; fi
if [ $rawstate == "ready" ]; then rawstate="0"; fi
# now, last-ditch cleanup - make sure that there is at least one number or "." in
# the value
# Note, this will fail on values that contain two dots and will blow up the import.
# like a value that is "3.2.4.2" will make it through to this point
# Stuff like dates (10/10/2016) or serial numbers will get passed through here
# and any non-numeric, non "." will be stripped out. Ugly, very ugly.
#
value=`echo $rawstate | grep [0-9.]`
# OK, strip out anything that isn't 0-9 or "."
value=${rawstate//[^0-9.]/}
if ! [ -z "$rawstate" ];
then
echo $cleandevice",host=$site,creator=indigo6.1.4,type=dev value="$rawstate >> $tmpdir/influx-out
fi
fi
done
IFS=$oldIFS
}
# Fetch the variables from Indigo
#
function getvariables {
IFS=$oldIFS
# get the variables file
cd $tmpdir;$wgetcmd1/variables.txt
# first, fetch all the variable values from the Indigo server, ensuring
# that no stupid spaces screw things up
oldIFS=$IFS
IFS=$'\n'
for variable in `cat $tmpdir/variables.txt`
do
variablename=`echo $variable | cut -f 3 -d "/"`
IFS=$oldIFS
cd $tmpdir/variables;`$wgetbin -q $wgetuser "$wgethost$variable"`
done
# Next iterate through all those files and search for stuff
#
IFS=$'\n'
for variable in `ls $tmpdir/variables`
do
value=`grep value $tmpdir/variables/$variable|tr -d " "|cut -f 2 -d :`
if ! [ -z "$value" ];
then
# clear up weird characters, spaces in measurement name
cleanvariable=${variable//[^a-zA-Z0-9_-]/}
cleanvariable=`echo $cleanvariable | rev | cut -c 4- | rev`
# now, clear up values to be float.
# If you have special strings that you want to match and
# set to 0 or 1 or whatever, this is the place to do it.
# Note that booleans are not supported. Feel free to patch.
if [ $value == "off" ]; then value="0"; fi
if [ $value == "on" ]; then value="1"; fi
# now, last-ditch cleanup - make sure that there is at least one number or "." in
# the value
# Note, this will fail on values that contain two dots and will blow up the import.
# like a value that is "3.2.4.2" will make it through to this point
# Stuff like dates (10/10/2016) or serial numbers will get passed through here
# and any non-numeric, non "." will be stripped out. Ugly, very ugly.
#
value=`echo $value | grep [0-9.]`
# OK, strip out anything that isn't 0-9 or "."
value=${value//[^0-9.]/}
if ! [ -z "$value" ];
then
echo $cleanvariable",host=$site,creator=indigo6.1.4,type=var value="$value >> $tmpdir/influx-out
fi
fi
done
IFS=$oldIFS
}
# now, run the main routines that call back to our functions, and post to influxdb
getvariables
getdevices
# Post to InfluxDB with the final result file
$curl -v -i -XPOST 'https://'$isite':'$iport'/write?db='$idb'&precision=s&u='$iuser'&p='$ipass'' --data-binary @$tmpdir/influx-out