System Crash

Posted on
Mon Jan 02, 2017 10:13 am
bkmar1192 offline
Posts: 274
Joined: Sep 12, 2015

System Crash

I am having an odd periodic issue with my Indigo server. It sort of crashes...

From the server itself:
- The Indigo client stops responding
- The indigo server shows that it is running
- All of the plugin hosts show 0.1 cpu usage and the server is really low cpu (see attached screen shot)
- Forcing quit on the client and restarting I can not reconnect to the server
- Forcing quit and restarting the server does not fix it - same issue

From remote computer:
- I am able to connect/stay connected
- everything says it is still running
-I get an error about reflector no longer responding
- I get an error about an embedded script not finishing (not sure if this is a cause or result)

Anything I can do to track down what is happening?
Attachments
CPU.png
CPU Usage on the server
CPU.png (366.44 KiB) Viewed 3190 times
Error.png
Error on client
Error.png (91.66 KiB) Viewed 3190 times

Posted on
Mon Jan 02, 2017 1:16 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: System Crash

When you connect from the remote Mac, are things semi-functional? Can you control devices?

Is the only way to get it working again to restart the Mac itself?

The next time it occurs, try unplugging USB hardware (Z-Stick, PowerLinc, etc.). Does that get Indigo to be responsive again?

Image

Posted on
Tue Jan 03, 2017 9:59 pm
bkmar1192 offline
Posts: 274
Joined: Sep 12, 2015

Re: System Crash

OK, just happened again.

matt (support) wrote:
When you connect from the remote Mac, are things semi-functional? Can you control devices?


Nothing is working. Everything shows still connected but nothing works and no output in the logs.

matt (support) wrote:
Is the only way to get it working again to restart the Mac itself?


That is the only way that I have found so far. Killing the Indigo server process and restarting just puts me back to the same place. Seems that there is another process that is foobarred but not sure what other processes to kill.

matt (support) wrote:
The next time it occurs, try unplugging USB hardware (Z-Stick, PowerLinc, etc.). Does that get Indigo to be responsive again?


No - no effect.

Scheduled scripts are still running.

I reviewed the logs and most of the plugins are still logging to their respective log files. However, sql-logger, iTunes, home bridge, ActionCollection, and device collection all stopped logging at 6:45. Not sure if this means anything or not. None of the logs seems to have any incriminating evidence in them.

I could turn off all of my plugins and slowly turn them on but the time between crashes seems to be pretty inconsistent.

I was able to close all plugins but got the following each time: process (pid 1137) failed to quit after polite request -- forcing it to quit now.

After quitting zwave, I tried restarting it but it won't start up.

Posted on
Wed Jan 04, 2017 6:05 pm
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: System Crash

This:

bkmar1192 wrote:
Nothing is working. Everything shows still connected but nothing works and no output in the logs.


would seem to be in direct contradiction to this:

bkmar1192 wrote:
Scheduled scripts are still running.

I reviewed the logs and most of the plugins are still logging to their respective log files. However, sql-logger, iTunes, home bridge, ActionCollection, and device collection all stopped logging at 6:45. Not sure if this means anything or not. None of the logs seems to have any incriminating evidence in them.

I could turn off all of my plugins and slowly turn them on but the time between crashes seems to be pretty inconsistent.

I was able to close all plugins but got the following each time: process (pid 1137) failed to quit after polite request -- forcing it to quit now.

After quitting zwave, I tried restarting it but it won't start up.


Are the scripts that are running doing anything?

So, unfortunately, I think the next step is to disable all 3rd party plugins and see if this happens again. If it does, the next thing would be to disable any scripts and try again.

If it doesn't fail with all 3rd party plugins disabled, then you have a smaller set of things to try.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Mon Jan 09, 2017 8:23 am
bkmar1192 offline
Posts: 274
Joined: Sep 12, 2015

Re: System Crash

Over the weekend I noticed that at the top of the hour there was a significant CPU spike in the Indigo server from 2% to 60-70%). It is the opposite eof the issue above but I thought maybe after a couple of these instances the indigo server went totall footer. Assuming the two issues were related I spent most of yesterday turning off scripts and plugins and thought I had found the culprit (a scheduled script I use to check the status of security devices).

I have nothing that runs at the top of the hour but the security script runs every second so I assumed there was some combination of stuff that happens at the top of the hour.

However, this morning the whole system was crashed again. I am at my wits end and very frustrated . So now I am not sure if I have 2 issues or if they are related and the script was not the issue.

I have turned off most non-essential plugins. The plugins that I have left on are either ones that I have used for a long time or turning them off would have an impact to my system and I would prefer to not turn them off.

I'd like to avoid a complete system rebuild but I am almost to that point...

This is what I know:
a) there is a cpu spike the Indigo server every hour
b) periodically (can't find any pattern) the Indy server and all plugins will go to 1% cpu
c) script executor stops responding and is constantly restarted
d) reflector stops responding and it continues to try and reconnect
e) some scripts seem to still run but all devices are non-responsive
f) web access is non-responsive
e) I am able to connect to Indigo from a remote computer
f) the local computer can't connect
g) the only way to recover is to reboot

Posted on
Mon Jan 09, 2017 8:44 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: System Crash

bkmar1192 wrote:
I have turned off most non-essential plugins. The plugins that I have left on are either ones that I have used for a long time or turning them off would have an impact to my system and I would prefer to not turn them off.

While I understand the desire to not turn off all plugins, I would suggest disabling all 3rd party plugins (individually, or in groups) for troubleshooting. That really is going to be the only way to determine if it is a plugin causing the problem.

Do you use Indigo's built-in email scanning? If so, temporarily disable the scanning (via the Preferences->Email panel, Check email every N minutes checkbox) Does that prevent the problem?

Lastly, the next time the problem occurs, what does Indigo show around the progress bar on the bottom-left of the Main Window? Is it empty or full of green? And is there any text above the progress bar describing what Indigo is doing?

Image

Posted on
Mon Jan 09, 2017 9:10 am
bkmar1192 offline
Posts: 274
Joined: Sep 12, 2015

Re: System Crash

I am not using email scanning.

I have turned off all plugins except airfoil pro and iTunes.

Here are the other plugins I run in case that is helpful:

AirFoil Pro
Cynical Cache
Homebridge Buddy
fingscan
Hue Lights
INDIGOPlotD
iTunes
Messages
Meta Devices
myQ Garage Door
NOAA Weather
Onkyo Network Remote Plugin
Security Camera
SQL Logger
TCPConnect (personal custom plugin for controlling TCPConnect light bulbs)

Posted on
Tue Jan 17, 2017 8:28 pm
cuhouse offline
Posts: 144
Joined: Feb 21, 2007
Location: Virginia, USA

Re: System Crash

Bert,
I am curious if you came up with a resolution to your server issue? I am experiencing very similar troubles and I am monitoring at this time to see if there is a particular plugin conflict. We have several plugin installs in common so I have started by disabling them first. I was hoping your lessons learned may get me to a solution quicker :)
Thanks,
Jody
Indigo 7
Late 2014 Mac Mini
10.12.2

Posted on
Tue Jan 17, 2017 8:36 pm
bkmar1192 offline
Posts: 274
Joined: Sep 12, 2015

Re: System Crash

I have not had a crash for about 48 hours now. I was just getting ready to turn a couple plugins on. Here are the ones that I have turned off right now.

1) Homebridge Buddy
2) Fingscan
3) IndigoPlotD
4) SQL Logger

I am beginning to think that it is a combination of plugins versus just a single plugin. This makes a bit more difficult to track down. Let me know if you find anything.

Posted on
Tue Jan 17, 2017 10:03 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: System Crash

I can talk about my plugins:

FINGSCAN can do this if you have
1. set the sleep parameter too low (1 second) in config
2. use ping for down devices
3. if you have devices that have ip number changes happening frequently (eg apple TV, most samsung TV) that are connected via wifi AND ethernet
What then happens it will update the device states too often and indigo CLIENT and Server and SQLlogger are kept busy updating each other.

==> set fingscan loop time to 5 secs
==> if you have ip devices that change IP numbers frequently disable duplicate updates in devices edit for these devices.

2. IndigoplotD with MATPLOT can use a lot of cpu time once every 5 minutes, but that is a simple straight forward single thread use cpu time issue and will not involve program to program communication that can lead to things waiting for each other - what this look like to me.


Karl

Posted on
Thu Jan 19, 2017 5:51 am
midd offline
Posts: 372
Joined: Apr 18, 2010

Re: System Crash

Try disabling only HomeBridge Buddy and see if there are any changes. I'm thinking that's the reason why my server has been stopping intermittently.

Indigo 7, Monterey (12.1) on a 2009 Mac Pro..

Posted on
Thu Jan 19, 2017 8:41 am
bkmar1192 offline
Posts: 274
Joined: Sep 12, 2015

Re: System Crash

I ran with PlotD and SQL logger as the only things turned off and I've been ok for a couple of days. When I turned on Fing scan I reduced the frequency of checks.

Last night I had a weird application memory issue that has never happened before. For now I am going to assume that was anomaly and unrelated.

I just turned plotd and logger back on so all of my plugins are running again - I'll report back in another 24 hours.


Sent from my iPhone using Tapatalk

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 5 guests