FileIOError

I never saw this happen before and it just hit me. Only other interesting thing that happened today is that Indigo lost connection to the PowerLinc briefly, even though we didn't lose power.

Code: Select all

2022-12-16 17:09:00.156   Power Failure   possible power failure
2022-12-16 17:09:02.555   Application   Connected to PowerLinc 2413U interface on port usbserial-A600B7SK
2022-12-16 17:09:02.556   Application   PowerLinc address 41.EF.64, firmware version 9.14
2022-12-16 17:09:02.604   Interface Failure   lost connection with interface
2022-12-16 17:09:02.605   Trigger   Power Failure

Then followed a few hours later by (non error lines removed):

Code: Select all

2022-12-16 20:20:19.708   Error   FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:21:00.338   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:22:00.401   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:23:00.419   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:24:00.399   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:25:00.380   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:26:00.360   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:27:00.382   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:28:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:28:09.625   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:29:00.419   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:30:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:30:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:31:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:32:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:33:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:34:00.418   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:35:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:36:00.419   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:37:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:38:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:38:10.020   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:39:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:40:00.431   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:41:00.421   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:42:00.421   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:43:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:44:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:45:00.425   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:45:00.426   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:46:00.425   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:47:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:48:00.423   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:48:10.173   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:49:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:50:00.425   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:51:00.425   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:52:00.423   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:53:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:54:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:55:00.423   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:56:00.426   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:57:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:58:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:58:10.325   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 20:59:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:00:00.402   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:00:00.403   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:01:00.401   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:02:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:03:00.422   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:04:00.417   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:05:00.397   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:06:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:07:00.357   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:08:00.416   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:08:10.471   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:09:00.417   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:10:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:11:00.417   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:12:00.417   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:13:00.415   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:14:00.416   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:15:00.423   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:15:00.424   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:16:00.415   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:17:00.416   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:18:00.413   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:18:10.619   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:19:00.416   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:20:00.414   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:21:00.414   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:22:00.414   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:23:00.420   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:24:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:25:00.412   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:26:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:27:00.416   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:28:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:28:10.768   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:29:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:30:00.412   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:30:00.412   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:31:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:32:00.416   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:33:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:34:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:35:00.416   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:36:00.411   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:37:00.406   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:38:00.408   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:38:10.880   Error   _DispatchProcessThread() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:39:00.409   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:40:00.405   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:41:00.385   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:42:00.364   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:43:00.415   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:44:00.365   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:45:00.325   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:45:00.326   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:46:00.367   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2022-12-16 21:47:00.340   Error   TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)

I had it happen again last night - same thing. Restarting the server fixed it, but I don't know why this has suddenly started happening - my setup has been pretty stable for years.

[jay (support)]

We suspect that there is some drive issue that's causing this issue since we can't reproduce it and it seems to be happening to a very small number of users. Have you specifically run any disk utilities to see if there are any issues with your disk? Is it a mechanical drive?

I'm running on a Mac Studio using its internal SSD, which has passed all tests and is not throwing any errors in the console log.

I am wondering if this is a ulimit resource limit that is being run into (perhaps a slow file handle leak?) What is the file or directory that is trying to be created? Is there additional debugging I could enable in case this happens again?

Another random thought- perhaps some maintenance process is purging temporary directories that Indigo is relying on? Just throwing out random ideas.

Part of it may be the sheer number of scripts I am running. I run a python script once a minute that polls energy usage from zwave outlets. When I start getting these errors, all the scheduled events fail with that error, as well as triggers and action groups.

Here is the script I am running every minute (somewhat redacted):

Code: Select all

#!/opt/homebrew/bin/python3

import time
import subprocess

ts = time.strftime('%Y-%m-%d %H:%M:00')

devices = {
    'a': 275332687,
    'b': 810484132,
    'c': 1489389006,
    'd': 1917954767,
    'e': 330980171,
    'f': 1138981718,
    'g': 682841392,
    'h': 77823954,
    'i': 1054359280,
    'j': 829018200,
    'k': 1426863970,
    'l': 1202758331,
    'm': 1098025627,
    'n': 1541504713
}

for id in devices.values():
    indigo.device.statusRequest(id, suppressLogging=True)
    time.sleep(1)

time.sleep(3)

values = []
for name, id in devices.items():
    current = indigo.devices[id].energyCurLevel / 120
    values.append("('current-" + name + "','" + ts + "'," + str(current) + ",0)")

sql = "-eINSERT INTO redacted VALUES " + ','.join(values) + ";"

subprocess.call(["/opt/homebrew/bin/mysql", "-uredacted", "-predacted", "-hredacted", "redacted", sql])


[matt (support)]

Unfortunately, we haven't been able to trace down what is causing this. It might be some type of resource leak, but so far we haven't been able to prove that (the number of open files / ports shown in the Activity Monitor reported by others that have seen the problem aren't getting super high). The problem does seem to be related to executing scripts, so it might be a leak inside the python interpreter Indigo runs. That said, I've spent a lot of time trying to reproduce the problem (running scripts every couple of seconds, etc.) and have never been able to reproduce the problem here. Therefore, I think it might also be system (not just Indigo) configuration specific.

I am starting to see this error now too... just upgraded to latest OS (13.1). Likely related but just after it started the denon-av could not be found. ?? Restarted resolved it. As a temp work around, is there anyway to recognize the state and auto restart?

2023-01-02 07:30:49.138 Schedule trigger "Wake Up - Turn on Sonos AMPs" (delayed action)
2023-01-02 07:30:49.471 Sent INSTEON "SONOS - Upper Landing - Bottom" on
2023-01-02 07:30:50.137 Schedule trigger "Wake Up - Turn on Sonos AMPs" (delayed action)
2023-01-02 07:30:50.479 Sent INSTEON "SONOS - Pool - Bottom" on
2023-01-02 07:30:51.138 Schedule trigger "Wake Up - Turn on Sonos AMPs" (delayed action)
2023-01-02 07:30:51.358 Sent INSTEON "SONOS - Garden - Top" on
2023-01-02 07:30:55.944 Error TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2023-01-02 07:30:59.923 Online Sensor "10.0.1.19" on
2023-01-02 07:31:00.139 Error TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2023-01-02 07:31:09.142 Schedule trigger "Master Bedroom TV On" (delayed action)
2023-01-02 07:31:09.142 Error unable to execute action -- org.cynic.indigo.denon-av not found or installed[/size]
2023-01-02 07:31:10.138 Schedule trigger "Wake Up" (delayed action)
2023-01-02 07:31:10.139 Error TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2023-01-02 07:31:11.961 Error TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
2

Hit it again today... Nothing obvious going on just before it occurs. It then appears that scheduled events still work and it's mainly impacting the ability to execute triggers. Any chance if there is a way to restart the server programmatically ? I could then perhaps set up a hourly schedule to restart it that is delayed/cancelled by a regular trigger (fired based on any Insteon command for example).

Jan 13, 2023 at 6:29:36 AM
Received INSTEON "ThermoStat (Lower)" temperature changed to 49.0

Jan 13, 2023 at 6:34:01 AM
Error FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)

Jan 13, 2023 at 6:35:00 AM
Error TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)
Error TriggerAction() caught exception: FileIOError -- failed to create file ref path -- getCatInfo failed (file or directory not found)

[matt (support)]

I don't think it will be possible to Trigger off of it given that Indigo / macOS kernel is in a bad state and not allowing Indigo to create any more file references. We are looking into ways to work around the problem currently, but this is a difficult one for us since we have yet to be able to reproduce the problem here.

Thanks for the quick reply. In my log the scheduled events seem to still fire, but if behind the scenes they try to create a file I can see that will be an issue too. Anyway, I coded up a hack that relies on triggers not working and will see if it works or not. Alternatively, if you could just not catch the exception and let it crash, I have launchd set up to restart it. Actually, in general, it might be a nice user option to offer a setting to relaunch on a crash (limited/throttled perhaps). Thanks again.

[matt (support)]

We believe we have fixed (avoided) this macOS bug by changing all of the file I/O APIs that Indigo Server were using in v2022.2, which is now available.

THANK YOU Matt and team for fixing this annoying bug!! The new version is working great!

[matt (support)]

Great – thanks for the confirmation report!