Plugin stops logging every midnight since upgrade

I recently moved my Indigo install to a newer Mac, and upgraded a bunch of stuff in the process, including this plugin. When it works it's fine but I have noticed that every night just after midnight local time it throws an error and stops logging:

Aug 15, 2019 at 12:10:35 AM
Grafana Home Dashboard Error error while trying to write to InfluxDB... please wait a moment, will work silently to correct in the background
Aug 15, 2019 at 12:14:59 AM
Grafana Home Dashboard Error error while connecting to InfluxDB after 12 attempts, please check configuration. Most recent connection error: HTTPSConnectionPool(host='cloud.nathanw.com', port=8086): Read timed out. (read timeout=5)

In the morning, if I restart the plugin, it starts logging again.

As you can see from those messages, I'm logging to an external InfluxDB install, and using SSL with it.

I'll try enabling more debugging before I go to bed tonight, but any thoughts in the meantime?

Plugin version 1.0.12
Indigo version 7.3.1
OSX 10.14.6

And you are sure that your external Influx is not going down every night at midnight?

My guess is either the retention policy is causing a disconnect, or simply the plugin is timing out with it’s connection.

It’s possible that both the above scenarios should be resolved with a simple reconnection. Perhaps I have a bug in the reconnect logic for external instances of Influx, I can try to take a look, but this is not my personal setup so I cannot test it myself.

Will try to take a look this weekend, sorry for the troubles.


Sent from my iPad using Tapatalk Pro

I'm pretty sure InfluxDB isn't going down because the other thing I have logging to it (my GEM power meter, via btmon.py) continues uninterrupted; I see the POST requests in my server log from it, but not from Indigo.

I did notice that the plugin has fairly old versions of Requests and urllib3 embedded. Any chance of getting those upgraded, in case a relevant bug in one of them got fixed? I admit I don't really know what the dependency situation is.

I'm wondering if something is going on with my connectivity. It appears that after ~5 total minutes of being unable to reach the InfluxDB server, it'll quit for good?

That's what I would expect in the event that it cannot reach your server. From memory, the plugin should attempt to connect a certain number of times, with pauses in between. I was wondering if this logic was not working properly for external connections. Once the number of connection attempts has been reached, it stops attempting to reconnect. I dont have the total attempt times, or the durations of the pauses handly. I need to review my comments and code to confirm.

Do you have a behavior you would like to see happen? Are you looking into why the connectivity is being broken / unable to succeed?

Regarding the Requests library, I may test removing it from the plugin and using the OS version instead, since it's a common package.

[jay (support)]

Regarding the Requests library, I may test removing it from the plugin and using the OS version instead, since it's a common package.

We've been including requests with Indigo since Indigo 7. I don't believe it's installed with the OS.

Regarding the Requests library, I may test removing it from the plugin and using the OS version instead, since it's a common package.

We've been including requests with Indigo since Indigo 7. I don't believe it's installed with the OS.

Cool, thanks.

On the disconnection issue - I hadn't previously suspected I had a connectivity problem, but I also don't know what else would notice. I'll see if I can set up some high-resolution monitoring tonight.

It seems a bit problematic, though, as lots of things could cause network disconnection for that long, like a power outage (which wouldn't stop Indigo since it's running on a laptop). I'd kind of expect exponential backoff of some kind, capping at something like hourly retries. Deciding to quit retrying for good means that manual intervention is going to be necessary after some interval, and that doesn't seem like a great thing.

That makes sense and I generally agree. For the majority of the users of the plugin, when using my built-in servers, failure connecting after the number of tries that I built in usually indicates a configuration problem and that the servers will not start up, which is why I perpetually quit connection attempts.

I'll look into adding an option in the config or if using an external InfluxDB, I'll do hourly retries. Will look this weekend.

Can you try the beta release that I am pushing to github now?

The behavior to expect is that that upon a connection error, it will display full error details. It will then attempt silently every minute for 10 attempts. After that, it will display a error and begin 15 minute interval connection attempts.

When using the built in InfluxDB server, it will stop after 12 attempts. When using an external server, it will not stop.

Sure, I'll try to grab that today.

(Last night I tried just changing the retry parameter from 12 to 120; I got an initial error at 12:10AM, a reconnect at 12:19, an error at 12:20, and then a reconnect at 12:26. The ping tests I was running to the same cloud host didn't show any dropouts, though.)

OK, upgraded. We'll see what that does tonight.

In the meantime, though, the configuration panel seems to have gotten a little wonky - some merge issues?
I don't remember (or understand) why there's both "InfluxDB port", and"InfluxDB HTTP port".

Influx consists of both the dB and the rest api server. When using external server it uses the http port to connect to the remote instance of influx. 8086 is the default.

The instructions on the config interface for the dB port is incorrect. That is ignored when using a external dB. I’ll correct that.

No changes to the config.


Sent from my iPhone using Tapatalk Pro

OK, it seemed to work overnight. Still more disconnects than really make sense to me, but at least it retried and managed to reconnect:

Aug 25, 2019 at 12:10:18 AM Grafana Home Dashboard Error error while trying to write to InfluxDB... please wait a moment, will work silently to correct in the background
Aug 25, 2019 at 12:17:04 AM Grafana Home Dashboard connecting to InfluxDB... cloud.nathanw.com:8086 using user account: indigo
Aug 25, 2019 at 12:17:19 AM Grafana Home Dashboard Error error while connecting to InfluxDB, will continue to try silently in the background.
Aug 25, 2019 at 12:23:35 AM Grafana Home Dashboard Error error while connecting to InfluxDB after 11 attempts, will now attempt silently every 15 minutes. Most recent connection error: HTTPSConnectionPool(host='cloud.nathanw.com', port=8086): Read timed out. (read timeout=5)
Aug 25, 2019 at 12:38:51 AM Grafana Home Dashboard Error error while connecting to InfluxDB after 12 attempts, will now attempt silently every 15 minutes. Most recent connection error: HTTPSConnectionPool(host='cloud.nathanw.com', port=8086): Read timed out. (read timeout=5)
Aug 25, 2019 at 12:53:51 AM Grafana Home Dashboard ######## connected to InfluxDB successfully... plugin will now resume logging data to InfluxDB ########