Grafana seems to need restarting periodically

Posted on
Sat Jan 05, 2019 9:43 am
dduff617 offline
Posts: 659
Joined: Jul 05, 2006
Location: Massachusetts, USA

Grafana seems to need restarting periodically

In my recent use and experimentation with Grafana over the last year or so, I find that if Grafana has been running for a while, it sometimes becomes unresponsive -- an attempt to connect to the Grafana URL will simply timeout.

If I select Plugins>Grafana Home Dashboard>Rebuild and Restart Grafana Server (Will not delete data), then everything seems to go back to normal after 10-15 seconds.

I have maybe 10-15 dashboards defined. I am not yet using alerting features of Grafana. Anyone else seeing Grafana die after being left running for a while? Any suggestion of what I can do to prevent it from happening? Any better way to fix it other than restarting Grafana? Is there any "harm" in restarting Grafana other than just the inconvenience of having to notice it not working and then perform the restart? The worst thing I've observed is if I make some tweaks to a dashboard design, and then leave it running but unsaved for a while and come back to it, there is a chance when I'll lose my changes I've made by needing to restart.

Posted on
Sun Jan 06, 2019 10:28 am
vtmikel offline
Posts: 628
Joined: Aug 31, 2012
Location: Boston, MA

Re: Grafana seems to need restarting periodically

Hey-

A few thoughts:

Glad to see you've already narrowed it down to Grafana, and not Influx as the problem. I assume you do not see any missing data from being collected?

There is no harm in restarting Grafana. Grafana is the component that requires a the least amount of computing resources in the plugin package. From my experience building the plugin, I've learned that Grafana is extremely lightweight. It does almost nothing "server side". The server mostly gives the web client the javascript to do most of the graphing on the client. The server stores configuration and passes through data (more on that below), and that's about it.

Can you confirm that you are running the latest version of the plugin and that your Grafana version is 5.3.4? Did this happen with prior version or is it recent? 5.4.2 is now out and I'll likely update the plugin to that version.

You may want to try to not leave any browser windows with Grafana up, to see if the Grafana server still becomes unresponsive despite having no clients open. What is your typical time range that you are viewing, you could also shorten that, in particular when you are leaving the browser open for long periods of time?

A second idea is to change the Data Souce from "Access" mode "Server" to "Browser".

The way that the plugin works is that the Indigo plugin is a data source to InfluxDB, which stores the data. Grafana is a client to the InfluxDB. As I mentioned, Grafana is heavily client based, but when "Access" mode for a data source is set to "Server", your web browser talks to the Grafana Server, which queries Influx. Changing the "Access" mode to Browser cuts the Grafana Server out of this process, and your browser connects to InfluxDB directly.

The refresh is typically every 10 seconds, so every 10 seconds or so your Grafana Server is refreshing the data from Influx. Depending on your Panels, and the way you have set up your plugin to send data from Indigo, you could have a lot of data going between the different components. If it was a resource issue, I'd be suprised to see that Grafana is the first to suffer, and you dont see problems with your InfluxDB or Indigo Server process for the plugin. My best guess at this point is that it's a number of open connections issue, which Grafana should be able to handle, but perhaps isn't. We could see if the update to Grafana fixes it (once I update the plugin) and take it from there.

Posted on
Sun Jan 06, 2019 11:51 am
dduff617 offline
Posts: 659
Joined: Jul 05, 2006
Location: Massachusetts, USA

Re: Grafana seems to need restarting periodically

vtmikel wrote:
Glad to see you've already narrowed it down to Grafana, and not Influx as the problem. I assume you do not see any missing data from being collected?


I'd hesitate to say I've definitively narrowed it down, but evidence does point to Grafana hanging. It occurred to me that it is possible, for example, that there could be some periodic process (could be Influx) that's bogging down and causing my attempt to access Grafana web UI to slow down or break, and that simply waiting 30 seconds may have the same effect as as restarting Grafana. Unlikely, but possible. I'll try to dig for more definitive data, including process/CPU states, etc when the problem occurs.

I haven't seen definitive evidence of missing data, although to be honest, I haven't looked carefully.

vtmikel wrote:

Can you confirm that you are running the latest version of the plugin and that your Grafana version is 5.3.4? Did this happen with prior version or is it recent? 5.4.2 is now out and I'll likely update the plugin to that version.


Confirmed - I am running 5.3.4. I believe I was seeing a similar problem on the prior version as well.

I sometimes access my GHD dashboards from outside my home LAN, so would only reluctantly switch Access Mode from Server to Browser as (if I understand correctly) this would require me to tunnel Influx traffic through my firewall - certainly doable, but something I'd prefer to avoid.

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 1 guest

cron