IndigoServer Crash 2021.1.0

Posted on
Sun Jul 25, 2021 12:12 am
davinci offline

Re: IndigoServer Crash 2021.1.0

Thanks!

I could rule out Hue Lights plugin.

After the restart I get this message which I assume is the faulty plugin process:
Stopped plugin process (pid 37849)

But no idea which one it is and there is no way to know that since the processes have no indication which plugins they run.

I am also running Prime95 torture test to check for RAM.

Do I have to test the Homebridge plugins as well or are they independent?
Should I test the reflector connection as well?

Posted on
Sun Jul 25, 2021 1:06 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: IndigoServer Crash 2021.1.0

If you open the Event Log file you should be able to search on 37849 and find which plugin that was. It'll be logged like this when the plugin was started:

Starting plugin "XYZ" (pid 37849)

I'm not positive the plugin Indigo is having to stop is the culprit, but if it consistently matches a particular plugin that could be a clue.

It could be related to the HomeBridge plugin. Less likely to be the reflector since there aren't many configuration options involved with those (so if that were the problem we would be hearing more reports of it).

Let us know how the memory test goes.

Image

Posted on
Mon Jul 26, 2021 2:51 am
davinci offline

Re: IndigoServer Crash 2021.1.0

The memory test was running for a few hours without any errors.

I will check on the next restart for the PID that will be stopped.

For now I disabled Pushover only and it didn't crash anymore. I will report back if this changes or I can even identify this plugin to crash the server. But looks good for now.

Homebridge ist still active.

Reflector was restarting last night, but I guess this could just be because of a bad internet connection.

Code: Select all
2021-07-26 01:57:11.948   Error   reflector connection test failed: Reflector connection is currently down.
2021-07-26 01:57:11.949   Warning   reflector reconnection scheduled in 5 seconds
2021-07-26 01:57:18.157   Reflector   restarting reflector connection to xxx

Posted on
Mon Jul 26, 2021 11:53 am
davinci offline

Re: IndigoServer Crash 2021.1.0

Pushover was not the cause for this crash. :)

Code: Select all
2021-07-26 16:40:32.412   Application   Starting interface "Z-Wave 7.5.1" (pid 430)
2021-07-26 16:40:45.518   Application   Starting plugin "RFXCOM 2.1.16" (pid 505)


Code: Select all
   Starting Indigo Server version 2021.1.0 on macOS Version 11.5 (Build 20G71)
   Loading database "/Library/Application Support/Perceptive Automation/Indigo 2021.1/Databases/Haus.indiDb"
   Stopped plugin process (pid 430)
   Stopped plugin process (pid 505)


Looks like the two USB devices are part of the problem...

So this issue might be related to M1 and macOS, because I had Kernel Panics when using the internal USB for a HDD, therefore I was using an adapter for USB-C (and a Hub for the interfaces/HDD) as many other users suggested. Next thing I will try is to use the normal USB just for the interfaces. Hopefully no kernel panic and no crashes.

Posted on
Tue Jul 27, 2021 9:36 am
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: IndigoServer Crash 2021.1.0

In modern Macs, kernel panics almost always indicate a hardware problem of some kind.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Tue Jul 27, 2021 12:01 pm
davinci offline

Re: IndigoServer Crash 2021.1.0

I agree but this is a well known M1 issue. The replacements have the same issue.

Posted on
Thu Aug 12, 2021 11:58 pm
davinci offline

Re: IndigoServer Crash 2021.1.0

Short update on the issue:

Hardware seems to be ok.
It still happens on 2021.1 and macOS 11.5.1 - which was to expect.

I have a LOT of python scripts running on triggers and schedules. Most of them are linked scripts. For now I changed all embedded (I hope I found all) to external scripts.

Is it possible that the crashes are triggered by scripts (internal or external)?

Posted on
Fri Aug 13, 2021 10:08 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: IndigoServer Crash 2021.1.0

Yes, it is possible since Indigo runs python scripts (embedded and external linked) inside of an IndigoPluginHost process which calls into the Indigo Server (which is what crashes while trying to respond to a command or request).

Image

Posted on
Fri Aug 13, 2021 1:36 pm
davinci offline

Re: IndigoServer Crash 2021.1.0

But external scripts are the recommended way? Or is ist the same? If I remember correctly it is handled differently.

Is there a way to find the script that causes this?

Btw. no crashes since all scripts are external links. Hope it stays like that. Maybe the embedded scripts are just too much to handle...

Posted on
Sat Aug 14, 2021 2:02 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: IndigoServer Crash 2021.1.0

davinci wrote:
But external scripts are the recommended way? Or is ist the same? If I remember correctly it is handled differently.

External scripts each run in their own process, which can help prevent blocking of the main Indigo Server and if the plugin host process crashes it won't take down the server.

However, in this case the main Indigo Server process is crashing, but it is crashing while trying to complete a command / request coming from a script or plugin (based on the crash call stack).

davinci wrote:
Is there a way to find the script that causes this?


Unfortunately no. The crash call stack just doesn't give that level of detail, and I think the crash is likely occurring a bit after this mishap. Like a car that gets off the road and then later crashes into a tree, all we see is the car hit a tree but have no idea why it left the road. I really don't know of a way to track this one down without the painstaking process of disabling scripts / plugins until the problem stops. I wish that weren't the case, as I know this is a pain...

Image

Posted on
Sat Aug 14, 2021 2:12 pm
davinci offline

Re: IndigoServer Crash 2021.1.0

Thanks a lot for the details.

I now changed all the scripts to external. Maybe I was using the embedded scripts too often at once.
For now everything seems to work an I couldn‘t reproduce the crash. Hope that was it.

Posted on
Mon Aug 16, 2021 9:56 am
davinci offline

Re: IndigoServer Crash 2021.1.0

It still crashes. :cry:

Posted on
Tue Oct 26, 2021 1:31 pm
davinci offline

Re: IndigoServer Crash 2021.1.0

Problem is still here on 12.0.1.

However it will restart after reboot maybe once and then stays stable for weeks.

This might also show why most of us have no issues, because if you miss the first crash it just works.

Posted on
Tue Nov 09, 2021 5:57 am
davinci offline

Re: IndigoServer Crash 2021.1.0

It never crashed since the last update to 2021.1.2.

So if you did something that could have fixed it, thanks.

Posted on
Tue Nov 09, 2021 11:08 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: IndigoServer Crash 2021.1.0

I don't believe there were any specific changes that would fix this, but we also never were able to track down exactly what was causing it. So fingers crossed it did get fixed.

Image

Who is online

Users browsing this forum: No registered users and 3 guests