INDIGO plot --> see new version INDIGOplotD

Posted on
Mon Feb 17, 2014 11:55 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

INDIGO plot --> see new version INDIGOplotD

please use new version posted at: http://forums.indigodomo.com/viewtopic.php?f=134&t=11983

EDITED:


INDIGO-PLOT
Version 0.7.3
Feb 23, 2014
Karl Wachs
Use as you see fit, no warranties


THIS IS STILL BETA, you should expect that some things don't work 100% and that data structures might change (you will need to reconfigure your parameters if they change)
If you use it and like it, let me know what could be improved or where you see issues / bugs.


Who is the audience:
===================
Users that would like to use INDIGO to automatically create plots/graphs of eg temperature, humidity, energy used, AC-ON time etc . vs time.
Low and medium level computer literacy should be ok. Highest technical challenge is the installation of "gnuplot".
If you write your own programs you might want more options in terms of data collection or plot parameters etc.
Although the expert could also add raw gnuPLOT commands in the menu

After install it should take about 5 minutes to see your first plot - NO PROGRAMMING needed.
Start easy with a simple plot and then do more complicated things. Easy way to screw things up is to use the wrong y axis ranges etc..


What it does:
============
It uses indigo device data and creates PNG plot files that can be used to be shown on iPhone, iPad, web etc
All configuration is done through menus.

The data shown is plotted over time:
- 2-day plot with 5 minute average/sum/count/min/max of data (592 bins)
- 14-day plot with hourly average/sum/count/min/max of data (336 bins)
- 390-day plot with daily average/sum/count/min/max of data (390 bins)
at midnight the data shifts to the day before (ie one day to the left in the graph)

All plots are created in 2 configurable sizes for e.g. iPhone or iPad.
Up to 10 lines (solid/dashed/filled boxes and width 0-6) or solid histograms can be shown in one plot
Each line, histogram and background can have one of 15 different colors
Each line or histogram can be assigned to the left or right axes
Both Y axes parameters can be set: range, scale, label

What does it not do / allow / missing
============================
-gnuPlots / graphs can have infinite number of parameters, size, shape, colors, foreground, back ground. We have selected a number of options that the plugin supports.
-Only the default font can be used.
-Deleting device & properties is not available. (version 1.+)
- we have selected only certain types of devices to collect data from i.e. thermostats, temp, energy, NOA, rain devices. Each device has some specifics that have to be taken into account. We have not included simple switches or dimmers or outlets. if there is demand for them we can add them.
- would like to add a math function: (a) multiply value by constant to e.g. convert from Watt to kWatt -- that should be easy.
(b) In addition to be able to add 2 properties e.g. sum up all energy used from all devices..

How to install:
===============
PREREQUISITE: GNUPLOT is installed. You can use homebrew or fink .. to install the package. It takes about 10 minutes.

install homebrew and gnuplot:
_____________________________

1. paste into a terminal window - this will install MAC’s Xcode installation tool:
xcode-select --install

you need to click on <install> the “command line tools”

2. paste into a terminal window to install homebrew:
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

3. paste into a terminal window to finally install gnuplot:
brew install gnuplot

4. then download the indigoplot plugin from the indigo site and double click on it to install it.


How to use it:
==============
after install, enable plugin in indigo and goto the menu: Plugin/INDIGOplot/config and setup your plots:


PART ONE: define the data you like to use — check the box <setup Devices / First Step> (uncheck the others, this is for convenience to make the window smaller)
________

you first select the data sources i.e. devices and properties you like to plot ( e.g. Device: thermostatUpstairs with property: temperature)

you can select 8 properties from each of 20 different devices (current limit, can be changed) to collect data from:
- Device Number 1-20: select a device number, new or existing

- Select Device: select a device from the presented list

- <CONFIRM> click the confirm button <CONFIRM> to save

- 1. property: select a property from the presented list to be tracked
- data type select from average over bin-time, sum of valus over bin-tim, count (value > 0) …, min or max of value in bin time ( bin-time = 5 minutes, 1 hour, 1 day
-
- 8. property: up to 8 **)

- <CONFIRM>: click confirm button <CONFIRM>

repeat for each device you like to use data from.

**)
-on / off information is recorded as 0/1
- count records 0 if less or equal 0, otherwise 1
- sum adds up values, e.g. in 5 minutes 5 measurements are taken, the sum will be the sum over these 5 numbers
- min, max is the lowest or highest value during the measurement time
- currently only thermostat/temp/humidity/energy/rain/SensorValue-devices are supported (will be adding other device/properties when I get one or get the properties to be plotted)
- we use a nicer name for the properties i.e. accumEnergy would be Energy or energy is actually Power. We are using a translation table for the properties. .. if yo find a mistake, let us know…


PART Two: define the Plots you like to create
________

- un-check the box <setup Devices / First Step>
- check the box <setup Plots / Second Step>
(to make the window smaller)

for each plot repeat the next steps
- Line Number/command 1-10: select the Plot Number(1-20)

- Plot File Name: This is used for all the files (gnu, png). If you have 2 with the same name .. that will not work.. :-(

- Title text: enter title or leave blank if you don’t want one.

- Grid......on/off: 0 or 1: set grid on / off: enter “1” or “on: to set grid on

- Left Scale..... FROM:TO: enter Y range in format low:high, leave empty to use automatic range

-...Tick Marks eg 1,2,5,10: enter tick marks to be place on left Y axis, leave empty form automatic mode

-… Label: enter the text that goes on the left axis

- Right Scale..... FROM:TO: enter Y range in format low:high, leave empty to use automatic range

-...Tick Marks eg 1,2,5,10: enter tick marks to be place on right Y axis, leave empty form automatic mode

-… Label: enter the text that goes on the right Y axis

the right Y axes can be used to e.g. show % ON time or humidity in % while on the left scale you show eg temperature..
you might notice in the default that the range goes from 0-200 while the scale from 0-100 that will limit the lines to
the lower half of the plot for e.g. ON time (e.g. heat, AC, fan ..). Or if you set range to 0-300 the lines will be limited to the lower 1/3.

- Resolution for.. set 1 x,y: these are in # of pixels x,y for selected plot size 1

- ...set 2 x,y: same for set 2 size for selected plot leave empty to not produce plot at this size

- Background Color: select one of the offered colors

- Text Color: select one of the offered colors, make sure that the text is complementary to the background .. black on blue does not work well..

- raw gnuplot cmd …: skip the raw gnu plot command (leave it empty): for gnuplot experts only (see expert section at bottom)

- <CONFIRM> when you are done with the plot click on the button <CONFIRM>
then either define more plots (change plot number/ name ..) or continue to define the lines

- <DELETE>:… to delete a plot select the plot number and click on <DELETE>
this will delete all files ( ...png ...gnu ...) and index/text etc information


PART Three: define the Line for the Plots
________

- un-check the box <setup Plots / Second Step>
- check the box <setup Lines / Third Step>
to make the window smaller

Define the lines to be used:
- Line Number/command 1-10: select a new one or an existing one

- Data-Source for LineA: select a data source you defined in PART ONE

- Data-Source for LineB: If you want to add/subtract other data from lineA use LineB, e.g. 2 energy using devices, add them up as one line

- lineA-B math: Operation between LineA and LineB, chose "None" if only one device/property should be used

- gnuplot Raw cmd: skip the raw gnu plot command (leave it empty) for now.
This is an option for gnuplot experts (see expert section at bottom)
any text > 2 characters will mute all other line options

- Multiplier: enter multiplication factor and offset if desired, i.e. when data is in WATT and you like to display in kWATT use 0.001

- Offset: to shift the curve up use Offset

- Line Key: enter the key to be used in the plot for THIS line

- use left or right scale: chose left / right scale for THIS line. It will use the proper scale etc to draw THIS line

- Line Type: solid/dashed/filled boxes

- Line Width: line width 1-6 , 1-3 are good choice 6 is REALY thick

- Line Color: pick one of the offered

- <CONFIRM>: confirm your input !! do not forget!!
repeat for additional lines

- <DELETE> to delete a Line select the Line number and click on <DELETE>
this will delete all files ( ...png ...gnu ...) and index/text etc information

Once all data plots and lines are entered click on <Save> IF YOU CANCEL ALL IS LOST
You can save the data after PART ONE and enter config again or after each of the PLOTS are defined. BUT WE CURRENTLY DO NOT RECOMMEND TO SAVE IN THE MIDDLE OF A PLOT. .. there is some error checking missing.

After 2+ minutes you can then the check the directory /Users/yourname/Documents/INDIGOplot for files "PlotNAME-timeBinning-ss.png" .. with timeBinning = minute/hour/day and ss=s1/s2 (2 sizes). Click on one of them and you should see the graph popping up.

If you want to change eg the back ground of a plot go back into the config menu and change the background in the Plot section and <save> ..2 minutes later you should have the change.


Using the png files on your iPad:
_________________________________

to show them on an ipdad you should put
“file:/Users/yourname/Documents/INDIGOplot/Fname-minute.s1.png” (replace your name and Fname with the proper strings)
into the URL field on the indigotouch design page. You don’t need better than 1 minute upgrade frequency.



HAPPY PLOTTING of temperature, humidity, watts ..
==================================================



Other MENU options:
===================
there are several other menu options in Plugins/INDIGOplot .....
-WRITE file paths and name of gnuplot to logfile .. This is useful if you setup your indigotouch file to show the plot on your iPhone or ipdad. just copy and paste into the proper field.
-SAVE Device configuration data to file .. just to have a backup of you configurations (can be edited .. but only for experts)
-SAVE PLOT configuration data to file .. same for PLOT info
-RESTORE DEVICE configuration data .. if you want to go back to an old config
-RESTORE PLOT configuration data .. same for PLOTS
-RESET DEVICE configuration data to default .. reset everything to default
-RESET PLOT configuration data to default .. same for PLOTS
-RESET DATA collected from devices .. set all data to 0
-PAUSE DATA collection .. useful for maintenance if you want to restore, restart indigo . etc
-CONTINUE DATA collection .. obvious
-Shift Day .. if you skipped a day, i.e. indigo was off it will have some data index issues,
this might help. only for technically interested, very special, mostly for testing
-ReCREATE plot files from parameter file .. if you forgot to save .. make new GNU program files from your settings, rarely used

That's it for non-programming folks..




GNUPLOT EXPERT SECTION..
========================

SOME MORE DETAILS.. ... if you like to really look under the hood..


raw gnuplot commands:
_____________________
You have an option to add raw gnu plot commands in the header section(PLOT menu) i.e.
before the gnu data section and in the data section (LINES Menu) after the plot “data file” ..


Enter the gnu command into the fields in the PLOT or LINE menu sections. The strings will be taken as given and inserted into the .gnu file as they are.
a typical data command could look like:
“ using 1:3 with lines linetype 6 linewidth 2 linecolor rgb "#FF8000" title “Power[Watt]” axis x1y1”
where 1:3 is the first and 3 column in the …dat files — remember that the data starts in column 3

Here you could also add two columns and divide by 1000 with “using 1:($3/1000+$4/1000) with .. “ to get to e.g. kWatt instead of Watt from 2 energy measurement devices.


In the header section (PLOT) you can also use raw gnuPLOT command: use e.g.: “set key off” to switch off the line keys.


config files:
_____________
The device / plot information is stored in the Configuration-Device.cfg and -Plot.cfg files. You COULD edit these files directly.
Most parts should be self evident.
The only really tricky part are the “INDEX” fields. They are the pointer from the PLOT/Line to the Device/Property items. i.e. which device/property is used for a line to be plotted. The INDEX data is at the top of the Configuration-Device.cfg & .bak file. If you are not into indexes etc do not mess with it.

In order to change the configuration manually
- "pause" the program.
- "save" the device & plot information.
- Edit the .bak file as you see fit, ie add lines to plots, change colors, line parameters etc
- "restore" the configuration files
All names and values are delimited with “++;++” before and after.
Most lines in the plot file are gnu plot parameters. INDEX, ON/OFF and Name fields are used to manage data.


GNU command files:
__________________

All the .gnu files are the command files for gnuplot (in a terminal session gnuplot fName-minute-s1.gnu will run the command file).
They are created using the information from the plot and device .cfg files.



data files:
___________

minute.dat hour.dat and day.dat are the files that hold the data collected from the devices over time.
their structure is as follow:

for minute.dat:
201402150505 #ofmeasurements device/prop1 device/prop2 device/prop3 ...
201402150510 #ofmeasurements device/prop1 device/prop2 device/prop3 ...
for hour.dat:
2014021505 #ofmeasurements device/prop1 device/prop2 device/prop3 ...
2014021506 #ofmeasurements device/prop1 device/prop2 device/prop3 ...
for day.dat:
20140215 #ofmeasurements device/prop1 device/prop2 device/prop3 ...
20140216 #ofmeasurements device/prop1 device/prop2 device/prop3 ...

column 1 is the time stamp in minutes (first timestamp is: 2014 feb 15, 5:05 am), or hours or days
column 2 is the number of measurements taken in this time period.. used for averages
columms 3 .. are the averaged values of the selected devices / properties in the configuration menu.



other remarks:
______________

Please see the manuals or webpages for gnuplot to understand the commands if you want to use the raw commands


the color codes are taken from:
http://www.uni-hamburg.de/Wiss/FB/15/Su ... colors.htm
We can add more if you like, but I would like to limit the number to < 20.

a good gnuplot description is at
http://gnuplot.sourceforge.net/docs_4.2/node1.html


rgb color definitions implemented:
"#FFA500" orange
"#FF0000" light-red
"#A00000" dark-red
"#FFFF00" light-yellow
"#A0A000" dark-yellow
"#00F000" light-green
"#008000" dark-green
"#0000FF" light-blue
"#0000A0" dark-blue
"#8B00B2" light-indigo
"#4B0082" dark-indigo
"#000000" black
"#909090" dark-grey
"#E0E0E0" light-grey
"#FFFFFF" white

Checking the expert box will display detailed information:
1. “gnu“ commands fields show the data as it is used in the gnu file. You can use this if you want to compose your own plot commands in section 4" using 1:5 where the 5 would be the index+2.
2. "raw gnuplot cmd before plot data" : you can enter any valid gnuPlot command here. It will be executed just before the plot commands
3. "gnuplot Raw cmd" : you can enter a gunPlot "using x,y with .. command for the plot section
4. "resulting GnuPlot Command:" shows the resulting gnu plot line that will be inserted into the gnuPlot file.. for learning how this works.- Resolution 1,2 : these are in # of pixels x,y if you don't want to plot 2 resolutions put a "0" in one of them . That will suppress the resolution



for experts a typical gnuplot file:
Code: Select all
#!'/Users/karlwachs/Documents/INDIGOPlot/all-minute-S1.gnu'   
set output '/Users/karlwachs/Documents/INDIGOPlot/all-minute-S1.png'   
set terminal png enhanced medium size 800,350 dashlength 0.5     
set xdata time 
set timefmt x "%Y%m%d%H%M"   
set format x "%a\n%H:00"   
set yrange [00:100]   
set format y "%.0f"   
set ytics (0,20,40,60,80,100) nomirror   
set ylabel "      Temperature [F]"  textcolor  rgb "#FFFF00"   
set format y2 "%.0f"   
set y2range[-10:200]     
set y2tics (0,20,40,60,80,100)   
set y2label "Humidty and ON-Time[%]                ." textcolor  rgb "#FFFF00" 
set key inside center top horizontal Right noreverse enhanced autotitles nobox
unset grid
set border linecolor  rgb "#FFFF00" 
set key textcolor  rgb "#FFFF00" 
set title "Downstair Temperature Settings" textcolor   rgb "#FFFF00" 
set obj 1 rectangle behind from screen 0,0 to screen 1,1
set obj 1 fillstyle solid 1.0 fillcolor  rgb "#000000" 
###  here goes the raw command if entered 
plot '/Users/karlwachs/Documents/INDIGOPlot/minute.dat'\
  using 1:(($4*0.1+5.0) > 5.0 ? ($4*0.1+5.0) : 1/0)  with  fillsteps fill solid      linewidth 1    linecolor  rgb "#FF0000"   title "HEAT-ON"  axis x1y2\
 ,''   using 1:(($11*0.1+0.001) > 0.001 ? ($11*0.1+0.001) : 1/0)  with  fillsteps fill solid      linewidth 1    linecolor  rgb "#0000FF"   title "AC-ON"  axis x1y2\
 ,''   using 1:(($3*1.0+0.0) )  with  lines       linetype 6    linewidth 2    linecolor  rgb "#008000"   title "Inside..temp"  axis x1y1\
 ,''   using 1:(($10*1.0+0.0) )  with  lines       linetype 6    linewidth 2    linecolor  rgb "#FFFFFF"   title "..Humidity"  axis x1y2\
 ,''   using 1:(($7*1.0+0.0) )  with  lines       linetype 0    linewidth 1    linecolor  rgb "#00F000"   title "Outside..Temp"  axis x1y1\
 ,''   using 1:(($8*1.0+0.0001) > 0.0001 ? ($8*1.0+0.0001) : 1/0)  with  lines       linetype 0    linewidth 1    linecolor  rgb "#FFFFFF"   title "..Humidity"  axis x1y2\
 ,''   using 1:(($5*1.0+0.0) )  with  lines       linetype 0    linewidth 1    linecolor  rgb "#0000FF"   title "setpoint..Cool"  axis x1y1\
 ,''   using 1:(($6*1.0+0.0) )  with  lines       linetype 0    linewidth 1    linecolor  rgb "#FF0000"   title "..Heat"  axis x1y1\
,''   using 1:($2-$2)                                                  with  lines       linetype 0    linewidth 1   linecolor   rgb "#000000"    title " "  axis x1y1
Attachments
step3.tiff
step 3
step3.tiff (103.08 KiB) Viewed 8307 times
step2.tiff
step 2
step2.tiff (153.72 KiB) Viewed 8307 times
step1.tiff
step 1
step1.tiff (137.59 KiB) Viewed 8307 times
plot1.tiff
example plot created with 3 devices thermostat, humidity sensor and outdoor temp/humidity sensor
plot1.tiff (58.14 KiB) Viewed 8307 times
Last edited by kw123 on Fri Jun 06, 2014 2:31 am, edited 14 times in total.

Posted on
Tue Feb 18, 2014 3:59 am
lalid offline
Posts: 23
Joined: May 04, 2012

Re: INDIGO plot

NICE!

" We have not included simple switches or dimmers or outlets. if there is demand for them we can add them. "

YES! Yes,there is a demand to check dimmers and outlets. And consider that some Z-wave devices does report engergy status.
It would be nice to be able to have grafs on total energy usage and per outlet :D

Posted on
Tue Feb 18, 2014 7:57 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: INDIGO plot

anything that has the attached properties is included. So yes the zwave switches are .. just not "simple" switches with no energy info..

"hvacHeaterIsOn"
,"hvacCoolerIsOn"
,"hvacFanIsOn"
,"setpointHeat"
,"setpointCool"
,"temperatureInput1"
,"hvacDehumidifierIsOn"
,"hvacHumidifierIsOn"
,"temperature"
,"humidity"
,"accumEnergyTotal"
,"curEnergyLevel"
,"temperatureC"
,"temperatureF"
,"rainrate"
,"currentDayTotal"

Posted on
Tue Feb 18, 2014 8:01 am
RogueProeliator offline
User avatar
Posts: 2501
Joined: Nov 13, 2012
Location: Baton Rouge, LA

Re: INDIGO plot

One possible suggestion for the future that might could support nearly any device would be to allow users to define data sources from SQL Logger's data. This would involve choosing the columns from a table after a device had been selected and then the data is obtained via a simple query. That is how I was going to create a graphing plugin but I had been stalling on the hope that someone would beat me to it... looks like my strategy paid off! ;-)

Thanks for letting us know, look forward to checking it out!

Posted on
Tue Feb 18, 2014 11:54 am
Korey offline
User avatar
Posts: 811
Joined: Jun 04, 2008
Location: Henderson, NV

Re: INDIGO plot

Looking forward to this!!

Thanks!

:D :D

--
Korey

Posted on
Tue Feb 18, 2014 2:59 pm
Lunkan offline
Posts: 13
Joined: Aug 18, 2013
Location: Sweden

Re: INDIGO plot

Yes :D This is exactly what I'm missing in Indigo. Can't wait, looking forward to this plugin.

Posted on
Tue Feb 18, 2014 6:40 pm
Indy_Larry offline
Posts: 133
Joined: Dec 03, 2013
Location: Indianapolis

Re: INDIGO plot

This is a nice addition to the Indigo family. I'll be checking this out soon. Thanks for your work on this project and sharing it with us.

Posted on
Tue Feb 18, 2014 7:24 pm
artpics offline
Posts: 232
Joined: Feb 24, 2009
Location: Calabasas CA

Re: INDIGO plot

3. then download the indigoplot plugin from attached link and double click on it to infall it.



where do i find the link to download the plugin

Posted on
Tue Feb 18, 2014 8:17 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: INDIGO plot

tomorrow.. still testing

Posted on
Tue Feb 18, 2014 8:37 pm
artpics offline
Posts: 232
Joined: Feb 24, 2009
Location: Calabasas CA

Re: INDIGO plot

Thought so. Looking forward to this plugin will it support interactive graphs?
thanks for plugin

INDIGO-PLOT
Version 0.6
Feb 19, 2014
Karl Wachs

Posted on
Tue Feb 18, 2014 9:56 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: INDIGO plot

no sorry

here the functionality
1. configure
2. run in background
3. produce updated plots every minute

Posted on
Wed Feb 19, 2014 8:02 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: INDIGO plot

v0.6.0 released:

https://www.dropbox.com/s/agzwyvhvzu2cla5/INDIGOplot-0-6-0.zip

please send comments and feedback.

hope this works for you

Karl

Posted on
Wed Feb 19, 2014 12:37 pm
mat offline
Posts: 769
Joined: Nov 25, 2010
Location: Cambridgeshire - UK

Re: INDIGO plot

thanks for this, just what i was looking for....

loaded it up very quickly, and set up as per instructions but not getting any png files appearing. I can see the dat files are updating every minute.

sorry not going to have much time over the next few day, but let me know what you need to start troubleshooting.

looking forward to this.

Mat

Late 2018 mini 10.14

Posted on
Wed Feb 19, 2014 1:32 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: INDIGO plot

did you install gnuPlot?

to test it if you have gnuPlot: open terminal and enter gnuplot
if you don't get a prompt gnuplot> you need to install gnu plot ... see the read me file on how to .

Karl

Posted on
Wed Feb 19, 2014 8:16 pm
artpics offline
Posts: 232
Joined: Feb 24, 2009
Location: Calabasas CA

Re: INDIGO plot

All working here

some questions.

i can only choose my nest thermostat under devices. How do i get my aoetec motion temp to show up. i also have many Variables for temp, ph, rain, etc how do i graph a variable.

other wise all working well. Thanks
Attachments
graph.jpg
graph.jpg (60.19 KiB) Viewed 8224 times

Who is online

Users browsing this forum: No registered users and 1 guest