InfluxDB plugin

Posted on
Tue Aug 15, 2017 2:33 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

I should update those, I've made things a bit more folder-aware since then.

Posted on
Thu Aug 17, 2017 1:40 pm
vtmikel offline
Posts: 628
Joined: Aug 31, 2012
Location: Boston, MA

Re: InfluxDB plugin

I had a good time playing with Grafana using the data from InfluxDB and this plugin yesterday. However, today I noticed some strange behavior, missing data in Influx, etc. I went to turn on debugging on your plugin and the dialog would not come up. After restarting and enabling debugging, I think I might be flooding my InfluxDB by the use of "Timed Devices" plugin, as well as my own custom timers. These devices are changing states every second, which I think is propagating to InfluxDB and basically crashing the plugin. Assuming my theory is correct, any way to filter which devices make it to InfluxDB?

Also, note that my InfluxDB is not on my local machine. I put it on a docker on my NAS, which isn't exactly a power house.

Mike

Posted on
Fri Aug 18, 2017 11:31 am
outsidenwnc offline
Posts: 24
Joined: Apr 20, 2014

Re: InfluxDB plugin

I get the error below trying to install on OSX 10.12.6 regarding the six 1.4.1 package -- Any workarounds or recommendations?
...
Downloading influxdb-4.1.1-py2.py3-none-any.whl (65kB)
100% |████████████████████████████████| 71kB 929kB/s
Requirement already satisfied: pytz in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from influxdb)
Collecting python-dateutil>=2.6.0 (from influxdb)
Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
100% |████████████████████████████████| 194kB 575kB/s
Collecting six>=1.10.0 (from influxdb)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting requests>=2.17.0 (from influxdb)
Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
100% |████████████████████████████████| 92kB 1.6MB/s
Collecting idna<2.7,>=2.5 (from requests>=2.17.0->influxdb)
Downloading idna-2.6-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 2.0MB/s
Collecting urllib3<1.23,>=1.21.1 (from requests>=2.17.0->influxdb)
Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)
100% |████████████████████████████████| 133kB 1.3MB/s
Collecting certifi>=2017.4.17 (from requests>=2.17.0->influxdb)
Downloading certifi-2017.7.27.1-py2.py3-none-any.whl (349kB)
100% |████████████████████████████████| 358kB 1.0MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.17.0->influxdb)
Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
100% |████████████████████████████████| 143kB 1.5MB/s
Installing collected packages: six, python-dateutil, idna, urllib3, certifi, chardet, requests, influxdb
Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 778, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 754, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-NhEvjf-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

Posted on
Mon Aug 28, 2017 1:15 pm
vtmikel offline
Posts: 628
Joined: Aug 31, 2012
Location: Boston, MA

Re: InfluxDB plugin

outsidenwnc,

I experienced this too, I used the --ignore-installed six flag and got around it. You'll have to open the script and run the commands manually.


smurfless1, I've made some modifications to your plugin for my setup. Maybe I have a larger database of devices than you do (specifically timers that update every second), but the plugin was unusable because of the number of device updates that Indigo was sending to your plugin for processing. I was even having trouble entering my Influx credentials because the plugin was crashing the configuration dialog.

My solution was that I created a list of default states that I wanted to always be sent to Influx, and added a way for individual devices to have states added using the Global Property Manager. The plugin ignore other updates. Let me know if you are interested in the code.

Mike

Posted on
Mon Sep 04, 2017 12:50 pm
bkmar1192 offline
Posts: 274
Joined: Sep 12, 2015

Re: InfluxDB plugin

Just an FYI. I tried installing your plugin and two things happened:

1) When trying to launch I get "Module not found"
2) The shell script broke my install of Pillow - messed up the jpg portion. I just needed to reinstall so not a big deal but might be something to look at or warn about.

Posted on
Tue Sep 05, 2017 12:26 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

Sorry folks, just back from cleaning things up in Houston. Not pretty.

I'll have a look. Python's world unfortunately isn't stable and predictable - a problem solved by many things we can't use in plugins. I have a backup plan, but it's not as clean as just modifying the system install of python.

The workaround for that would be to rewrite it so that the only thing the plugin does is write out to a file/queue/something that is NOT dependent on external python modules, and have an entirely separate python virtual environment where we can set it up JUST SO, and read from the queue of commands, publish to Influx, etc. Again, not as pretty, so I didn't pursue it. Yet. The major impact would be that you have to start the plugin, and then have something else that does the queue->influx work.

And I thought that I had a lot of devices with my 100ish. Yikes! Yeah, send me a PM with notes on how you set up the limiting.

Posted on
Tue Sep 05, 2017 1:01 pm
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: InfluxDB plugin

Is the problem that there are multiple python libraries wanting to use different versions of the same library/module? That one is a problem if you can't figure out which versions of each module use the same version of the shared module. That's certainly a problem. Unfortunately, it would take considerable effort for us to change the plugin mechanism to use a virtual environment (where you could more specifically control those sorts of things).

Note, however, that library version dependencies aren't new or specific to Python, and I don't think I agree that it makes Python unstable or unpredictable.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Mon Sep 11, 2017 2:35 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

I just phrased it wrong, I completely understand. In C++-land, where I spend my day job, you can statically link the libraries you want inside your binary to work around many dependency conflicts. In Python, I use a virtual environment to do the same thing, but it's always been a bit more work to do. Apparently a dependency is conflicting with other things, which is all I meant to point out. Because the system python is something that can be modified, it is a potential source of conflicts.

My current tinker-time has brought us a new plugin (not posted yet) that will simply use -no additional libraries- *gasp* to multicast out to your local machine the json version of a device or variable update. Then we'll start a separate process (or processes) that can listen to those updates, and load any libraries they want in their own virtual environment, and that should get us away from that particular problem and into all sorts of new ones!

Tonight I'll start porting the influx posting code from the original plugin to a new process, and think about how I might be able to make it less work to set up this whole Rube Goldberg machine. Does anyone know the rules/license problems related to sharing a pre-built virtualenv? If not I can go with the more traditional "run this script to set up your environment to run this" method.

Posted on
Wed Sep 13, 2017 1:58 pm
smurfless1 offline
User avatar
Posts: 38
Joined: Sep 23, 2015
Location: Round Rock, TX

Re: InfluxDB plugin

Part 1:
https://github.com/smurfless1/indigo-json-broadcaster

Broadcasts the same JSON as before, but requires no external python modules. It does open a port, so if your on-mac firewall gripes, that's why. This only broadcasts it to other processes that know how to listen to it, not to Influx.

Part 2:
https://github.com/smurfless1/indigo-influx-receiver

This is a python 2.x script, and is safe(st) to run from a virutualenv. Depends on the python influxdb module. Will pipe the JSON from the above plugin to Influxdb. Takes command line args to alter the default server/port/etc.

Later I'll perform the same magic on the Kafka plugin, but it seems it might be just me using that one.

Posted on
Fri Sep 29, 2017 12:04 pm
vtmikel offline
Posts: 628
Joined: Aug 31, 2012
Location: Boston, MA

Re: InfluxDB plugin

I've been incrementally building my dashboard for my home using Grafana. Thought I would share what I've built so far. The image is too big to attach, see it here: https://www.dropbox.com/s/wk3bk9urkncbvtj/Dashboard%20Picture.png?dl=0

Posted on
Mon Oct 02, 2017 2:58 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: InfluxDB plugin

vtmikel wrote:
I've been incrementally building my dashboard for my home using Grafana. Thought I would share what I've built so far. The image is too big to attach, see it here: https://www.dropbox.com/s/wk3bk9urkncbvtj/Dashboard%20Picture.png?dl=0

That looks very nice! Some great data visualization.

Image

Posted on
Wed Oct 04, 2017 12:05 pm
blysik offline
Posts: 213
Joined: Jan 06, 2015

Re: InfluxDB plugin

vtmikel wrote:
I've been incrementally building my dashboard for my home using Grafana. Thought I would share what I've built so far. The image is too big to attach, see it here: https://www.dropbox.com/s/wk3bk9urkncbvtj/Dashboard%20Picture.png?dl=0


That's awesome! I've started my own dashboard as well. (We use grafana at work, so it's already a product I love.)

Do you think you could share your dashboard json?

Posted on
Wed Oct 04, 2017 1:06 pm
vtmikel offline
Posts: 628
Joined: Aug 31, 2012
Location: Boston, MA

Re: InfluxDB plugin

blysik wrote:
vtmikel wrote:
I've been incrementally building my dashboard for my home using Grafana. Thought I would share what I've built so far. The image is too big to attach, see it here: https://www.dropbox.com/s/wk3bk9urkncbvtj/Dashboard%20Picture.png?dl=0


That's awesome! I've started my own dashboard as well. (We use grafana at work, so it's already a product I love.)

Do you think you could share your dashboard json?


In order to build the dashboard I attached, I had to modify the plugin that smurfless1 built for two reasons:

1. I have a lot of devices that have custom states that change frequently (like timers from this plugin: http://forums.indigodomo.com/viewtopic.php?f=134&t=18785). This was overloading Influx with updates that were not necessary. I wanted a way to control the properties that were being sent to Influx, so I built a mechanism into the plugin to filter with include and exclude lists.

2. I found that my visualizations were missing data on the left side (especially for discrete graphs) for devices / variables that are updated infrequently. For example, say my HVAC turns on yesterday at 11:50pm and turns off at 11:59pm. Then it turns on today at 10am until 10:10:am. If I graph today only, Grafana doesn't know what the HVAC state was from midnight to 10am. As far as I saw, there was no way in Grafana to solve this problem, so I added a ability to have a minimum update frequency sent to Influx so that my graphs were complete.

I posted my fork to the Influx plugin to Github: https://github.com/mlamoure/indigo-influxdb

I'm happy to share my dashboard JSON. I'm terrible at Grafana, so I doubt it will be very helpful as I've statically written most of my queries. Email me at mlamoure at gmail.

Posted on
Wed Oct 04, 2017 1:14 pm
blysik offline
Posts: 213
Joined: Jan 06, 2015

Re: InfluxDB plugin

vtmikel wrote:
In order to build the dashboard I attached, I had to modify the plugin that smurfless1 built for two reasons:

1. I have a lot of devices that have custom states that change frequently (like timers from this plugin: http://forums.indigodomo.com/viewtopic.php?f=134&t=18785). This was overloading Influx with updates that were not necessary. I wanted a way to control the properties that were being sent to Influx, so I built a mechanism into the plugin to filter with include and exclude lists.

2. I found that my visualizations were missing data on the left side (especially for discrete graphs) for devices / variables that are updated infrequently. For example, say my HVAC turns on yesterday at 11:50pm and turns off at 11:59pm. Then it turns on today at 10am until 10:10:am. If I graph today only, Grafana doesn't know what the HVAC state was from midnight to 10am. As far as I saw, there was no way in Grafana to solve this problem, so I added a ability to have a minimum update frequency sent to Influx so that my graphs were complete.

I posted my fork to the Influx plugin to Github: https://github.com/mlamoure/indigo-influxdb


These seem to be very reasonable enhancements to the plugin. Any word from the initial author about this?

Posted on
Fri Oct 13, 2017 10:35 pm
koensayr offline
Posts: 90
Joined: Jul 10, 2013

Re: InfluxDB plugin

Perfect! Thank you!

Who is online

Users browsing this forum: No registered users and 1 guest