INDIGO plot --> see new version INDIGOplotD

yes, 2.x works on pre mavericks.
just keep using gnuplot,
added matplot as option for some folks that had problems with gnu plot ---conflict with sonos?plugin.
On <10.9 you could install matplot yourself. They are 99% equivalent in the produced result

matplot: creates bigger files, uses more cpu, pictures look a little better some how (subjective) - and I learned a lot putting it in
gnuplot: stable, works, need to retro add the x label addition that were added with matplot (its really minor)

Karl

regarding smoothing the lines.. attached is a plot where I have smoothed 2 lines (with a bezier function). One is temperature (the upper 2) and windspeed from NOAA -the 2 lower ones - They are just shifted apart, should overlap, but to show the difference I have added an offset.

NOAA only updates its data every 2 hours and I believe it is the value at that point of time. Thats why the data looks as it is, but using a smoothing function somehow changes the nature of the data.

anyway: we could add a "smooth" option per line.. Any votes for that?


and here the code for that, just have to add "smooth bezier" to the end of the line (this is the code for the 2 lower lines in gnuplot)

Code: Select all

 ,''   using 1:((($35)*3.0) +15)    with  lines linetype 6   linewidth 3   linecolor rgb "#F0FF00"   title "as is"   axis x1y2 \
 ,''   using 1:((($35)*3.0) )       with  lines linetype 6   linewidth 3   linecolor rgb "#F000FF"   title "bezier"  axis x1y2 smooth bezier\


Karl

regarding smoothing the lines..

anyway: we could add a "smooth" option per line.. Any votes for that?



Karl

+1

anyway: we could add a "smooth" option per line.. Any votes for that?

Vote for smoothing!

https://www.dropbox.com/s/4h5klfr57tg57k2/INDIGOplot-2-2-0.zipv2-2-0

3 smoothing options
bezier, s-bezier, c-spline, the other options do not make sense, they require non zero data and some other constraints.

attached an example plot with the original, and the 3 smooth options of external temperature at my house.
top and bottom set of 4 lines are the same, only difference is that the top are shifted to show them independently and the bottom 4 are overlaid.


I guess just try it out:

switch "expert mode on" and choose smooth option in the Math... section.

ONLY FOR GNUPLOT... they come out of the box in GNUPLot -- matplot requires detailed programming.. need some time for that

Karl

and here the GNUPLOT code, just FYI

Code: Select all

       using 1:((($22)*1.0+0.0) )  with  lines linetype 6   linewidth 2   linecolor rgb "#AAAAAA"   title "No smooth"   axis x1y1\
 ,''   using 1:((($22)*1.0-5.0) )  with  lines linetype 6   linewidth 2   linecolor rgb "#F000FF"   title "s-bezier"    axis x1y1  smooth sbezier\
 ,''   using 1:((($22)*1.0-10.0) ) with  lines linetype 6   linewidth 2   linecolor rgb "#FF0000"   title "bezier"      axis x1y1  smooth bezier\
 ,''   using 1:((($22)*1.0-15.0) ) with  lines linetype 6   linewidth 2   linecolor rgb "#0000FF"   title "csplines"    axis x1y1  smooth csplines\
 ,''   using 1:((($22)*1.0-40.0) ) with  lines linetype 6   linewidth 2   linecolor rgb "#AAAAAA"   title "No smooth"   axis x1y1\
 ,''   using 1:((($22)*1.0-40.0) ) with  lines linetype 6   linewidth 2   linecolor rgb "#F000FF"   title "s-bezier"    axis x1y1  smooth sbezier\
 ,''   using 1:((($22)*1.0-40.0) ) with  lines linetype 6   linewidth 2   linecolor rgb "#FF0000"   title "bezier"      axis x1y1  smooth bezier\
 ,''   using 1:((($22)*1.0-40.0) ) with  lines linetype 6   linewidth 2   linecolor rgb "#0000FF"   title "csplines"    axis x1y1  smooth csplines\


+100 for smoothing;)

I get 99 more +'s since it was my suggestion

Thanks again!
~Lou

i have a problem in the config dialog.

if i try to make a minor mod to one of my plots (for example to change just the string used in the plot legend), i get an error upon hitting "Save" button that says "Error validating fields: hover over the red field labels above to see the error reason".

no field labels appear to be red. at this point, i tried a number of things to get out of this state, including deleting the most recent lines that i had added to plots, etc. (even though the plot was working correctly just before i re-entered the configuration dialog).

eventually, i stumbled on the apparent cause: at some point previously, i had exposed the "Expert tools" part of the dialog and changed the number of days reported in the for the MINUTE size bins (from 2 to 3). this change seemed to work fine when i made it.

so the way i was able to get out of the "stuck" dialog that was refusing to save was to re-expose the "Expert tools" dialog and re-do the same setting (MINUTE bins from 2 days to 3 days). after that, save worked fine and my plots behave as expected.

unclear if this is a "bug" or just something people might need to be aware of when modifying some of the plot parameters. i guess possibly this is why these are under "expert" settings - mess with them at your own peril.

clarification: i have to expose the "expert tools", then hit "CONFIRM" button. this works, even without actually having to change any of the expert settings. but i have to do this before the overall "Save" button at the bottom of the dialog will let me save and exit the dialog.

thanks will check.

should be fixed next release this weekend.

Karl

concerning the save .. error message. The logic does not return that error message.
The program (validateDeviceConfigUi) always returns True. (I am in denial phase .. )

dduff617: could you double check the error message and when exactly it occurs: click on save, cancel, ...? and copy the exact wording.

Thanks


==> question to Matt and or Jay:

if validateDeviceConfigUi always does:

Code: Select all

return (True, valuesDict)

can there be an error message: "Error validating fields: hover over the red field labels above to see the error reason"

how else can this error message be created?

Thanks

Karl

ok, here's sequence of events:

starting out, i have two working plots. the config dialog is not open. afaik, the plot system is "happy" at this point.
next, i open the config dialog and (just as an example), i set out to change the title text string of one of the working plots via the following steps:

1. plugins>indigoplot>Configure...
2. select modify an existing plot from the menu in section 2 of the configuration dialog.
3. change value of the "title text" string in the configuration.
4. hit "CONFIRM" button at the bottom of section 2.
5. hit "Save" at the bottom of the configuration dialog.
6. an error displays within the dialog saying "Error validating fields. Hover over the red field labels above to see the error reason". however there are no red field labels.
7. click the "expert tools" checkbox at the top of the dialog.
8. (without making any other changes) click the "CONFIRM" button at the bottom of the bin size settings section.
9. click "Save" button (again) at the bottom of the overall config dialog.
10. config dialog closes and everything is happy again.

let me know if you would like any additional details.

[matt (support)]

==> question to Matt and or Jay:

if validateDeviceConfigUi always does:

Code: Select all

return (True, valuesDict)

can there be an error message: "Error validating fields: hover over the red field labels above to see the error reason"

It shouldn't be possible. Based on dduff's description of steps I think he is in the plugin (not device) config dialog though?

yes, the problem i described happens in the plugin config.

confession: what little i know about using indigo plot i got from poking around a bit in this thread, or from poking around within the config dialog. are there actual docs? i was under the impression that indigoplot does not provide any device types, triggers, actions or anything like that. is that right?

it feels like it might help break up the configuration stuff into more manageable chunks by have a "global" plugin config and then distinct IOM object types for a plot, and another unique type for a line. that's how some other plugins seem to handle it - for example, "find my iDevices" plugin allows for creation of objects representing locations, iCloud login credentials, rectangular regions, etc. then the user creates actions, triggers, schedules, etc. that reference these base objects. some objects deepen on and refer to other objects - for example, an iDevice object must refer to an existing appleID object. there are only a few things you set up in the top-level plugin config. the rest of the config is manipulated via the various objects.

dduff617: very good observation! I actually have a version that uses the device framework to enter data. But it is about the same for the user. The common parameters are: debug on/off, and binning of plots - they would stay in the config section, The other 20++ fields have to be entered the very a same way. Actually to switch from the config view to entering the parameters in the device view is just one change in the program.

WHAT I WOULD LIKE TO SEE: a hierarchical menu structure. Right now it is done (here) by switch sections of the menu on and off in the background.


Karl

i'm running indigo on a machine that i recently updated to mavericks.

twice recently, i left the machine for a while and came back to find fans running and machine very very sluggish to respond. digging around, it looked like the CPU was pegged and memory usage was very high. it seemed that kernel_task was hogging memory (>9GB) and things were going downhill from there. first time, i just rebooted and things came back and seemed to be working ok.

second time this happened, i did more investigation. i gradually shut off other services running on the box (crash plan, etc.). cpu usage decreased but memory usage stayed very high until in indigo i tried Indigo>Plugins>indigoplot>Reload. at this point, the kernel_task memory usage dropped to around 1GB. so this suggests to me that something in the plugin or the accompanying plot package is triggering the problem. anyone else see this?

after i reload the plugin, the machine seemingly becomes "normal" again in terms of being responsive to clicks, etc. in this condition, cpu usage remains steady in single digits - nothing unusual there.

a few minutes later, i do notice that the biggest memory process is now "Python" which from the open files is clearly associated with indigoplot at around 2.75GB. i've been watching delta's for Python and kernel_task for a few minutes and they don't appear to be growing very fast.

more details:

right now (still definitely under "experimentation") indigoplot is configured with four devices (including weather, two thermostats, and a multi-sensor) and is generating four plots, each with 2-6 lines.

indigo version: 6.0.11
'python --version': 2.7.5
mac os x 10.9.2 (server) -- just upgraded, so i have little baseline experience.
matplotlib.__version__: 1.1.1

other indigo plugins i use:
find my iDevices
google voice sms
noaa weather plus
sonos
sql logger (set to use SQLite)
and a few others...

other processes on the machine:
crashplan, which itself tends to be a bit of a memory pig and obviously generates a lot of disk i/o.

https://www.dropbox.com/s/2dt1e69sg816tpx/INDIGOplot-2-3-0.zip v2-3-0

fixes:
- plotting min/ max values tracking was not correct

added, improved:
- configuration is snappier, added some performance improvements
- changed sql import to better manage larger data from devices like NOAA (it has 20+++ properties and each time anything changes it creates anew entry for each property( e.g. humidity, pressure..)
- added smoothing options in MATPLOT. It also doubles..quadruples bins if e.g. you have few bins. If you have "day plot" with e.g. 60 days/bin it uses splines to interpolate 240 bins, looks very nice now, see attached

dduff617: yes, MATPLOT uses MUCH more cpu than GNUPLOT and the png files are 4-5 times bigger than GNUPLOT. The new version(2-3-0) should reduce that CPU consumption. I have 4 plots with 6 lines and all smoothing options enabled and they are done in 4 seconds. The PYTHON process name that uses the CPU is the MATPLOT process. Mine *)goes up to 55% for ~ 3 seconds and comes down again to 0. What kind of server hardware are you using? When you run it (new version), could you check matplot.log in the indigoplot directory. It should show the seconds used to generate the plots. And thanks for the feedback.
I could also reduce the frequency for how often the plots get generated from 1 minute to 5 minutes. It currently checks if there is new data (data file timestamp has changed), waits 15 seconds and then generates the plots.

Karl

*) my server is a Mac Pro 2008 14GB RAM 4 CPU, 500GB SSD)

the attached plot shows the different options for MATPLOT smoothing. The numbers behind the options show the smoothing factor used in the program higher = more smoothing) the lines have been shifted down to show line each separately