Help me debug my daily Indigo web interface crash

Indigo is working for all functionality at present EXCEPT accessing from the app/web.

This happens periodically, but it happened three days in a row now.
Symptoms:
- I abruptly cannot connect to Indigo on the app, Domopad, or web
- Same problem on LAN as remote access via port forwarding
- the server is running on my mac, logging, responding to triggers, etc.
- Grafana’s server (via plugin) is running fine
- Restarting the server fixes the problem.

Now, my log is littered with information, and there’s no error that jumps out at me.

Any thoughts on how to diagnose this?


MunDMC

[jay (support)]

If you filter your log on the word Error, what do you get? You should do this directly on the log file itself and while the problem is happening.

If you filter your log on the word Error, what do you get? You should do this directly on the log file itself and while the problem is happening.

I’m sorry for not knowing what might/should be obvious, but what is the name (or path) of the log file you speak of?


MunDMC

I think the following may correlate with the problem- happened last night:

Code: Select all

May 17, 2019 at 3:54:00 AM
   Script                          House Mode: houseSleep

May 17, 2019 at 4:03:28 AM
   Domotics Pad Mobile Client      Process command: updateMobileDeviceStates

May 17, 2019 at 4:04:00 AM
   Schedule                        CLIMATE_anlaytics
   Action Group                    CONTROLPAGE_CLIMATE_analytics
   Script                          74.2 CP_TEMP_devon
   Script                          69.9 CP_TEMP_hall2
   Script                          73.7 CP_TEMP_hall3
   Script                          72.6 CP_TEMP_kitchen
   Script                          69.9 CP_TEMP_livingroom
   Script                          73.4 CP_TEMP_mbr
   Script                          67.2 CP_TEMP_den
   Script                          74.7 CP_TEMP_thea
   Script                          73.5 CP_TEMP_masterbath
   Script                          73.4 CP_TEMP_mbr

May 17, 2019 at 4:04:55 AM
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########
   Grafana Home Dashboard          shutting down the Grafana Server...
   Script                          House Mode: houseSleep

May 17, 2019 at 4:05:55 AM
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:06:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:07:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:08:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:09:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:10:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:11:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:12:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:13:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########
   Mac Apps Error                  Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 118, in runConcurrentThread
  File "plugin.py", line 273, in update
  File "plugin.py", line 341, in psInfo
  File "plugin.py", line 130, in psResults
  File "plugin.py", line 468, in do_shell_script
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1223, in _execute_child
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1175, in pipe_cloexec
OSError: [Errno 23] Too many open files in system

   Mac Apps Error                  plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)

May 17, 2019 at 4:14:55 AM
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:15:13 AM
   Domotics Pad Mobile Client      Process command: updateMobileDeviceStates

May 17, 2019 at 4:15:46 AM
   Timed Devices                   "TIMEDDEVICES_ACTIVITY_hall2" on
   Sent Hue Lights                 "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_3" on to 10 at ramp rate 0.0 sec.
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########
   Script                          House Mode: houseSleep
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   Online Sensor Error             "ONLINE_SENSOR_publicip" thread error: [Errno 23] Too many open files in system
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_681787073 ("TIMER_light_hall1") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_681787073 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_1118843866 ("TIMER_light_hall2") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_1118843866 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   SQL Logger Error                exception trying to insert row into table device_history_942040989 ("TIMER_light_hall3") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_942040989 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   Hue Lights Error                Failed to connect to the Hue bridge at 192.168.1.174. - Check that the bridge is connected, turned on and the network settings are correct.
   Sonos Error                     SOAPSend Error: HTTPConnectionPool(host='192.168.1.247', port=1400): Max retries exceeded with url: /MediaRenderer/AVTransport/Control (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1080f9f90>: Failed to establish a new connection: [Errno 23] Too many open files in system',))
   Sonos Error                     Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 47, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 381, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 633, in updateRelTime
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 707, in SOAPSend
UnboundLocalError: local variable 'response' referenced before assignment

   Sonos Error                     plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_1230827709 ("TIMEDDEVICES_ACTIVITY_hall2") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_1230827709 ("displaystate") VALUES (%s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_681787073 ("TIMER_light_hall1") -- unable to modify table
   SQL Logger Error                unable to open database file
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                INSERT INTO device_history_681787073 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_1118843866 ("TIMER_light_hall2") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_1118843866 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   SQL Logger Error                exception trying to insert row into table device_history_942040989 ("TIMER_light_hall3") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_942040989 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   Sonos                           SoCo Reactor Ignition...

May 17, 2019 at 4:16:39 AM
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Den]: avTransport subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Den]: renderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Den]: zoneGroupTopology subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Den]: queue subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Den]: groupRenderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Den]: contentDirectory subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Den]: groupManagement subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Garage]: renderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Kitchen]: avTransport subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Kitchen]: renderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Kitchen]: zoneGroupTopology subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Kitchen]: queue subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Kitchen]: groupRenderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Kitchen]: contentDirectory subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Kitchen]: groupManagement subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos One]: avTransport subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos One]: renderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos One]: zoneGroupTopology subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos One]: queue subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos One]: groupRenderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos One]: contentDirectory subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos One]: groupManagement subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Thea]: avTransport subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Thea]: renderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Thea]: zoneGroupTopology subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Thea]: queue subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Thea]: groupRenderingControl subscription failed.  Resubscribing...
   Sonos Error                     [Fri May 17 04:16:39 2019] [Sonos Thea]: groupManagement subscription failed.  Resubscribing...
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_1230827709 ("TIMEDDEVICES_ACTIVITY_hall2") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_1230827709 ("displaystate") VALUES (%s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_681787073 ("TIMER_light_hall1") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_681787073 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_1118843866 ("TIMER_light_hall2") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_1118843866 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   SQL Logger Error                exception trying to insert row into table device_history_942040989 ("TIMER_light_hall3") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_942040989 ("longstatusstring", "timeleftseconds") VALUES (%s, %s);
   Sonos Error                     SOAPSend Error: HTTPConnectionPool(host='192.168.1.247', port=1400): Max retries exceeded with url: /MediaRenderer/AVTransport/Control (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105cb33d0>: Failed to establish a new connection: [Errno 23] Too many open files in system',))
   Sonos Error                     Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 47, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 381, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 633, in updateRelTime
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 707, in SOAPSend
UnboundLocalError: local variable 'response' referenced before assignment

   Sonos Error                     plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_1188127001 ("MULTISENSOR MASTERBATH - Temperature") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_1188127001 ("sensorvalue_ui", "sensorvalue") VALUES (%s, %s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table variable_history_1796995897 ("DRIP_CURRENTURL_iphone")
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO variable_history_1796995897 (value) VALUES (%s);
   SQL Logger Error                exception trying to insert row into table variable_history_188568234 ("DRIP_CURRENTURL_den")
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO variable_history_188568234 (value) VALUES (%s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table variable_history_75340285 ("DRIP_MASTER_source_info")
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO variable_history_75340285 (value) VALUES (%s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_833697727 ("MULTISENSOR MASTERBATH - Humidity") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_833697727 ("sensorvalue_ui", "sensorvalue") VALUES (%s, %s);
   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   Group Trigger Error             Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 262, in deviceUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

   SQL Logger Error                exception trying to insert row into table device_history_1230827709 ("TIMEDDEVICES_ACTIVITY_hall2") -- unable to modify table
   SQL Logger Error                unable to open database file
   SQL Logger Error                INSERT INTO device_history_1230827709 ("displaystate") VALUES (%s);
   Sonos                           Loaded Playlists... [6]
   Sonos                           ZonePlayer: Sonos One, Save Queue: Indigo_RINCON_7828CAB4DDEA01400
   Sonos                           ZonePlayer: Sonos Kitchen, Save Queue: Indigo_RINCON_B8E937572FFE01400
   Sonos                           ZonePlayer: Sonos Den, Nothing in Queue to Save
   Sonos                           ZonePlayer: Sonos Thea, Save Queue: Indigo_RINCON_B8E937B9068601400
   Sonos Error                     UPNP Error: Invalid Queue Request
   Sonos Error                     Offending Command -> zoneIP: 192.168.1.251, soapRoot: /MediaRenderer, soapBranch: /Queue, soapAction: SaveAsSonosPlaylist
   Sonos                           ZonePlayer: Sonos Garage, Save Queue: Indigo_RINCON_B8E937BBEC6001400
   Sonos                           Loaded Playlists... [6]
   Sonos                           SoCo Reactor Ignition...
   Sonos                           Loaded Playlists... [6]
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########

May 17, 2019 at 4:17:34 AM
   Sent Hue Lights                 "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_3" on to 10 at ramp rate 0.0 sec.

May 17, 2019 at 4:17:55 AM
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard          ######## About to (re) start Grafana.  Please be patient while this happens. ########
   Sent Hue Lights                 "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_3" on to 10 at ramp rate 0.0 sec.

May 17, 2019 at 4:18:45 AM
   Sent Hue Lights                 "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
   Sent Hue Lights                 "HUE_hall_3" on to 10 at ramp rate 0.0 sec.
   Grafana Home Dashboard Error    [Errno 24] Too many open files
   Grafana Home Dashboard Error    [Errno 24] Too many open files



Both Grafana and Sonos are throwing a lot of errors. Shut them both down and see if the problem goes away.

[matt (support)]

Looks like macOS is throwing lots of "too many open files" errors, which means something (maybe Grafana plugin that is reporting the error, but could be other plugins, scripts, or apps) is opening a ton of files and possibly not closing them when they are done.

If restarting Grafana and Sonos doesn't help then try a complete Mac reboot. If the problem continues to happen then try running without the Grafana plugin enabled for a while to see if it is the culprit.

Thanks Matt and FD- I kinda love and rely on those, but I’ll give it a whirl.

Bizarre that Grafana web server continues to run


MunDMC

Doh- IWS just stopped working again with Grafana disabled. Logs to follow. I forget- what’s the easiest way to restart the server remotely?

So I am uncertain WHEN it went down today, and I'm curious if there is any good way to script somethign that will check when the web server stops responding.

I cannot tell what is a primary error, and what is a downstream error. The plugins getting the most error messages appear to be: SQL logger, Sonos, and Group Tiriggers

Code: Select all

2019-05-21 03:17:00.649   Script   73.1 CP_TEMP_devon
2019-05-21 03:17:00.655   Script   72.1 CP_TEMP_hall2
2019-05-21 03:17:00.658   Script   73.4 CP_TEMP_hall3
2019-05-21 03:17:00.662   Script   75.8 CP_TEMP_kitchen
2019-05-21 03:17:00.665   Script   72.5 CP_TEMP_livingroom
2019-05-21 03:17:00.666   Script   69.8 CP_TEMP_mbr
2019-05-21 03:17:00.670   Script   76.7 CP_TEMP_den
2019-05-21 03:17:00.674   Script   73.6 CP_TEMP_thea
2019-05-21 03:17:00.675   Script   70.8 CP_TEMP_masterbath
2019-05-21 03:17:00.675   Script   69.8 CP_TEMP_mbr
2019-05-21 03:17:05.417   Domotics Pad Mobile Client   Process command: updateMobileDeviceStates
2019-05-21 03:21:26.236   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.94 W
2019-05-21 03:25:00.644   Script   House Mode: houseSleep
2019-05-21 03:26:28.044   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.88 W
2019-05-21 03:31:28.057   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.99 W
2019-05-21 03:34:58.163   Domotics Pad Mobile Client   Process command: updateMobileDeviceStates
2019-05-21 03:36:00.598   Script   House Mode: houseSleep
2019-05-21 03:36:28.022   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.88 W
2019-05-21 03:41:30.074   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.95 W
2019-05-21 03:45:32.843   WebServer   http://localhost:8176/variable_url_redirect/redirect?name=DRIP_CURRENTURL_den load error: [Errno 54] Connection reset by peer
2019-05-21 03:45:33.899   WebServer   disconnecting from Indigo server
2019-05-21 03:45:33.899   Application   WebServer client disconnected
2019-05-21 03:46:00.560   Schedule   CLIMATE_anlaytics
2019-05-21 03:46:00.561   Action Group   CONTROLPAGE_CLIMATE_analytics
2019-05-21 03:46:00.567   Script   72.9 CP_TEMP_devon
2019-05-21 03:46:00.575   Script   72.1 CP_TEMP_hall2
2019-05-21 03:46:00.578   Script   73.1 CP_TEMP_hall3
2019-05-21 03:46:00.582   Script   75.8 CP_TEMP_kitchen
2019-05-21 03:46:00.586   Script   72.5 CP_TEMP_livingroom
2019-05-21 03:46:00.587   Script   69.8 CP_TEMP_mbr
2019-05-21 03:46:00.591   Script   76.1 CP_TEMP_den
2019-05-21 03:46:00.595   Script   73.6 CP_TEMP_thea
2019-05-21 03:46:00.596   Script   70.7 CP_TEMP_masterbath
2019-05-21 03:46:00.596   Script   69.8 CP_TEMP_mbr
2019-05-21 03:46:30.896   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.93 W
2019-05-21 03:47:00.567   Script   House Mode: houseSleep
2019-05-21 03:51:29.606   Sonos Error   SOAPSend Error: HTTPConnectionPool(host='192.168.1.247', port=1400): Max retries exceeded with url: /MediaRenderer/AVTransport/Control (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1074f0050>: Failed to establish a new connection: [Errno 23] Too many open files in system',))
2019-05-21 03:51:29.608   Sonos Error   Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 47, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 381, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 633, in updateRelTime
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 707, in SOAPSend
UnboundLocalError: local variable 'response' referenced before assignment

2019-05-21 03:51:29.608   Sonos Error   plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
2019-05-21 03:51:31.890   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.97 W
2019-05-21 03:51:35.038   Mac Apps Error   Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 118, in runConcurrentThread
  File "plugin.py", line 273, in update
  File "plugin.py", line 341, in psInfo
  File "plugin.py", line 130, in psResults
  File "plugin.py", line 468, in do_shell_script
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1223, in _execute_child
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1175, in pipe_cloexec
OSError: [Errno 23] Too many open files in system

2019-05-21 03:51:35.039   Mac Apps Error   plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
2019-05-21 03:51:35.357   Hue Lights Error   Failed to connect to the Hue bridge at 192.168.1.174. - Check that the bridge is connected, turned on and the network settings are correct.
2019-05-21 03:51:39.613   Sonos   SoCo Reactor Ignition...
2019-05-21 03:51:40.547   Group Trigger Error   Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

2019-05-21 03:51:40.547   SQL Logger Error   exception trying to insert row into table variable_history_1796995897 ("DRIP_CURRENTURL_iphone")
2019-05-21 03:51:40.548   SQL Logger Error   unable to open database file
2019-05-21 03:51:40.548   SQL Logger Error   INSERT INTO variable_history_1796995897 (value) VALUES (%s);
2019-05-21 03:51:40.550   Group Trigger Error   Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

2019-05-21 03:51:40.550   SQL Logger Error   exception trying to insert row into table variable_history_75340285 ("DRIP_MASTER_source_info")
2019-05-21 03:51:40.550   SQL Logger Error   unable to open database file
2019-05-21 03:51:40.551   SQL Logger Error   INSERT INTO variable_history_75340285 (value) VALUES (%s);
2019-05-21 03:51:40.663   Sonos Error   SOAPSend Error: HTTPConnectionPool(host='192.168.1.247', port=1400): Max retries exceeded with url: /MediaRenderer/AVTransport/Control (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1075935d0>: Failed to establish a new connection: [Errno 23] Too many open files in system',))
2019-05-21 03:51:40.664   Sonos Error   Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 47, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 381, in runConcurrentThread
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 633, in updateRelTime
  File "/Library/Application Support/Perceptive Automation/Indigo 7.3/Plugins/Sonos.indigoPlugin/Contents/Server Plugin/Sonos.py", line 707, in SOAPSend
UnboundLocalError: local variable 'response' referenced before assignment

2019-05-21 03:51:40.664   Sonos Error   plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
2019-05-21 03:51:50.664   Sonos   SoCo Reactor Ignition...
2019-05-21 03:56:33.942   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.92 W
2019-05-21 03:58:00.539   Script   House Mode: houseSleep
2019-05-21 04:06:34.712   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.91 W
2019-05-21 04:09:00.507   Script   House Mode: houseSleep
2019-05-21 04:11:33.945   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" energy total to 6.93 kWh
2019-05-21 04:11:34.763   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.97 W
2019-05-21 04:15:00.482   Schedule   CLIMATE_anlaytics
2019-05-21 04:15:00.482   Action Group   CONTROLPAGE_CLIMATE_analytics
2019-05-21 04:15:00.499   Script   72.9 CP_TEMP_devon
2019-05-21 04:15:00.507   Script   72.1 CP_TEMP_hall2
2019-05-21 04:15:00.512   Script   72.7 CP_TEMP_hall3
2019-05-21 04:15:00.512   Script   74.6 CP_TEMP_kitchen
2019-05-21 04:15:00.517   Script   72.5 CP_TEMP_livingroom
2019-05-21 04:15:00.517   Script   70.0 CP_TEMP_mbr
2019-05-21 04:15:00.522   Script   76.4 CP_TEMP_den
2019-05-21 04:15:00.527   Script   73.4 CP_TEMP_thea
2019-05-21 04:15:00.528   Script   70.9 CP_TEMP_masterbath
2019-05-21 04:15:00.529   Script   70.0 CP_TEMP_mbr
2019-05-21 04:16:34.744   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.90 W
2019-05-21 04:20:00.488   Script   House Mode: houseSleep
2019-05-21 04:21:34.752   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.91 W
2019-05-21 04:26:36.796   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.87 W
2019-05-21 04:31:00.442   Script   House Mode: houseSleep
2019-05-21 04:31:37.587   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.98 W
2019-05-21 04:36:38.628   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.97 W
2019-05-21 04:39:38.469   Timed Devices   "TIMEDDEVICES_ACTIVITY_devon" on
2019-05-21 04:42:00.410   Script   House Mode: houseSleep
2019-05-21 04:44:00.403   Schedule   CLIMATE_anlaytics
2019-05-21 04:44:00.404   Action Group   CONTROLPAGE_CLIMATE_analytics
2019-05-21 04:44:00.410   Script   72.9 CP_TEMP_devon
2019-05-21 04:44:00.415   Script   72.1 CP_TEMP_hall2
2019-05-21 04:44:00.418   Script   72.6 CP_TEMP_hall3
2019-05-21 04:44:00.419   Script   74.6 CP_TEMP_kitchen
2019-05-21 04:44:00.422   Script   71.6 CP_TEMP_livingroom
2019-05-21 04:44:00.427   Script   70.0 CP_TEMP_mbr
2019-05-21 04:44:00.431   Script   75.4 CP_TEMP_den
2019-05-21 04:44:00.434   Script   73.4 CP_TEMP_thea
2019-05-21 04:44:00.434   Script   70.9 CP_TEMP_masterbath
2019-05-21 04:44:00.435   Script   70.0 CP_TEMP_mbr
2019-05-21 04:46:38.614   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.93 W
2019-05-21 04:49:39.149   Timed Devices   "TIMEDDEVICES_ACTIVITY_devon" off
2019-05-21 04:53:00.378   Script   House Mode: houseSleep
2019-05-21 04:56:40.433   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.94 W
2019-05-21 05:01:41.474   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.86 W
2019-05-21 05:04:00.347   Script   House Mode: houseSleep
2019-05-21 05:06:42.477   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.96 W
2019-05-21 05:11:41.547   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" energy total to 6.94 kWh
2019-05-21 05:11:42.493   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.99 W
2019-05-21 05:13:00.313   Schedule   CLIMATE_anlaytics
2019-05-21 05:13:00.314   Action Group   CONTROLPAGE_CLIMATE_analytics
2019-05-21 05:13:00.320   Script   72.7 CP_TEMP_devon
2019-05-21 05:13:00.326   Script   70.3 CP_TEMP_hall2
2019-05-21 05:13:00.329   Script   72.4 CP_TEMP_hall3
2019-05-21 05:13:00.330   Script   73.5 CP_TEMP_kitchen
2019-05-21 05:13:00.333   Script   71.6 CP_TEMP_livingroom
2019-05-21 05:13:00.333   Script   70.1 CP_TEMP_mbr
2019-05-21 05:13:00.337   Script   74.5 CP_TEMP_den
2019-05-21 05:13:00.340   Script   73.1 CP_TEMP_thea
2019-05-21 05:13:00.340   Script   70.8 CP_TEMP_masterbath
2019-05-21 05:13:00.341   Script   70.1 CP_TEMP_mbr
2019-05-21 05:15:00.331   Script   House Mode: houseSleep
2019-05-21 05:16:43.268   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.92 W
2019-05-21 05:20:09.483   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:20:09.488   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall3" on
2019-05-21 05:20:09.533   Sent Hue Lights   "HUE_hall_3" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:20:28.386   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall2" on
2019-05-21 05:20:28.393   Sent Hue Lights   "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:20:28.410   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:20:28.425   Sent Hue Lights   "HUE_hall_3" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:20:53.771   Sent Hue Lights   "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:20:53.787   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:21:02.773   Trigger   TIMER_MOTION_livingroom
2019-05-21 05:21:02.775   Trigger   TIMER_MOTION_livingroom_expired
2019-05-21 05:21:02.785   Timed Devices   "TIMEDDEVICES_ACTIVITY_livingroom" on
2019-05-21 05:21:02.811   Z-Wave   sent "unassigned zwave 15" set brightness to 50
2019-05-21 05:21:02.842   Z-Wave   sent "unassigned zwave 15" off
2019-05-21 05:21:14.009   Timed Devices   "TIMEDDEVICES_ACTIVITY_kitchen" on
2019-05-21 05:21:43.277   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 9.95 W
2019-05-21 05:22:28.484   Sent Hue Lights   "HUE_hall_3" off at ramp rate 0.0 sec.
2019-05-21 05:22:53.556   Sent Hue Lights   "HUE_hall_1_5" off at ramp rate 0.0 sec.
2019-05-21 05:22:53.600   Sent Hue Lights   "HUE_hall_2" off at ramp rate 0.0 sec.
2019-05-21 05:23:40.880   Received INSTEON   "SWITCH_kitchen_recessed" off
2019-05-21 05:23:46.274   Sent Hue Lights   "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:23:46.297   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:24:04.552   Trigger   TIMER_MOTION_livingroom
2019-05-21 05:24:04.554   Trigger   TIMER_MOTION_livingroom_expired
2019-05-21 05:24:04.596   Z-Wave   sent "unassigned zwave 15" set brightness to 50
2019-05-21 05:24:04.628   Z-Wave   sent "unassigned zwave 15" off
2019-05-21 05:24:05.770   Sent Hue Lights   "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:24:05.798   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:24:25.265   Sent Hue Lights   "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:24:25.280   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:24:42.955   Received INSTEON   "REMOTE_hall1" off (button 5)
2019-05-21 05:24:42.956   Trigger   REPEATER_HALL1_button5_off
2019-05-21 05:24:42.980   Sent Hue Lights   "HUE_hall_1" off at ramp rate 0.0 sec.
2019-05-21 05:24:44.714   Received INSTEON   "SWITCH_livingroom2" off
2019-05-21 05:24:44.715   Trigger   if LIVING ROOM SWITCH <<off>> then LIVINGROOM FLOOR LAMP
2019-05-21 05:24:44.790   Z-Wave   sent "SWITCH_ZWAVE_livingroom_spotlight" off
2019-05-21 05:24:44.882   Sent Hue Lights   "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:24:44.896   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:24:45.258   Sent INSTEON   "SWITCHLINK_livingroom_floorlamp" off
2019-05-21 05:24:50.297   Z-Wave   received "SWITCH_ZWAVE_livingroom_spotlight" power load to 0.00 W
2019-05-21 05:25:00.767   Sent Hue Lights   "HUE_hall_1_5" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:25:00.780   Sent Hue Lights   "HUE_hall_2" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:25:00.806   Sent Hue Lights   "HUE_hall_3" on to 10 at ramp rate 0.0 sec.
2019-05-21 05:25:04.096   Timed Devices   "TIMEDDEVICE_ACTIVITY_masterbath" on
2019-05-21 05:26:00.305   Script   House Mode: houseSleep
2019-05-21 05:27:00.440   Sent Hue Lights   "HUE_hall_1_5" off at ramp rate 0.0 sec.
2019-05-21 05:27:00.487   Sent Hue Lights   "HUE_hall_2" off at ramp rate 0.0 sec.
2019-05-21 05:27:00.503   Sent Hue Lights   "HUE_hall_3" off at ramp rate 0.0 sec.
2019-05-21 05:29:45.115   Timed Devices   "TIMEDDEVICES_ACTIVITY_devon" on
2019-05-21 05:30:10.077   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall3" off
2019-05-21 05:34:04.748   Timed Devices   "TIMEDDEVICES_ACTIVITY_livingroom" off
2019-05-21 05:35:00.884   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall2" off
2019-05-21 05:35:04.864   Timed Devices   "TIMEDDEVICE_ACTIVITY_masterbath" off
2019-05-21 05:35:06.877   Timed Devices   "TIMEDDEVICES_ACTIVITY_kitchen" off
2019-05-21 05:37:00.257   Script   House Mode: houseSleep
2019-05-21 05:39:45.685   Timed Devices   "TIMEDDEVICES_ACTIVITY_devon" off
2019-05-21 05:42:00.230   Schedule   CLIMATE_anlaytics
2019-05-21 05:42:00.232   Action Group   CONTROLPAGE_CLIMATE_analytics
2019-05-21 05:42:00.238   Script   72.5 CP_TEMP_devon
2019-05-21 05:42:00.244   Script   70.0 CP_TEMP_hall2
2019-05-21 05:42:00.247   Script   72.2 CP_TEMP_hall3
2019-05-21 05:42:00.247   Script   73.5 CP_TEMP_kitchen
2019-05-21 05:42:00.250   Script   70.4 CP_TEMP_livingroom
2019-05-21 05:42:00.251   Script   70.1 CP_TEMP_mbr
2019-05-21 05:42:00.252   Script   74.6 CP_TEMP_den
2019-05-21 05:42:00.255   Script   73.1 CP_TEMP_thea
2019-05-21 05:42:00.255   Script   70.9 CP_TEMP_masterbath
2019-05-21 05:42:00.256   Script   70.1 CP_TEMP_mbr
2019-05-21 05:48:00.221   Script   House Mode: houseSleep
2019-05-21 05:51:39.513   Received INSTEON   "OPEN_CLOSE_back_door" off (heartbeat)
2019-05-21 05:59:00.189   Script   House Mode: houseSleep
2019-05-21 06:00:50.908   Timed Devices   "TIMEDDEVICES_ACTIVITY_thea" on
2019-05-21 06:03:05.147   Timed Devices   "TIMEDDEVICE_ACTIVITY_masterbath" on
2019-05-21 06:10:00.175   Script   House Mode: dayTime
2019-05-21 06:10:00.176   Trigger   House State
2019-05-21 06:10:00.451   Pushover   Pushover notification was sent sucessfully, title: House State, body: dayTime
2019-05-21 06:10:17.666   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall2" on
2019-05-21 06:10:37.146   Received INSTEON   "SWITCH_kitchen_recessed" on
2019-05-21 06:10:37.178   Timed Devices   "TIMEDDEVICES_ACTIVITY_kitchen" on
2019-05-21 06:11:00.148   Schedule   CLIMATE_anlaytics
2019-05-21 06:11:00.148   Action Group   CONTROLPAGE_CLIMATE_analytics
2019-05-21 06:11:00.154   Script   72.4 CP_TEMP_devon
2019-05-21 06:11:00.160   Script   70.0 CP_TEMP_hall2
2019-05-21 06:11:00.164   Script   71.9 CP_TEMP_hall3
2019-05-21 06:11:00.164   Script   72.7 CP_TEMP_kitchen
2019-05-21 06:11:00.168   Script   70.4 CP_TEMP_livingroom
2019-05-21 06:11:00.169   Script   70.1 CP_TEMP_mbr
2019-05-21 06:11:00.169   Script   74.7 CP_TEMP_den
2019-05-21 06:11:00.174   Script   72.8 CP_TEMP_thea
2019-05-21 06:11:00.174   Script   71.2 CP_TEMP_masterbath
2019-05-21 06:11:00.175   Script   70.1 CP_TEMP_mbr
2019-05-21 06:15:28.636   Timed Devices   "TIMEDDEVICE_ACTIVITY_masterbath" off
2019-05-21 06:20:18.445   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall2" off
2019-05-21 06:20:50.544   Timed Devices   "TIMEDDEVICES_ACTIVITY_thea" off
2019-05-21 06:21:00.126   Script   House Mode: dayTime
2019-05-21 06:21:40.425   Timed Devices   "TIMEDDEVICES_ACTIVITY_livingroom" on
2019-05-21 06:26:29.634   Timed Devices   "TIMEDDEVICES_ACTIVITY_thea" on
2019-05-21 06:30:47.078   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall3" on
2019-05-21 06:31:41.330   Timed Devices   "TIMEDDEVICES_ACTIVITY_livingroom" off
2019-05-21 06:32:00.094   Script   House Mode: dayTime
2019-05-21 06:40:00.068   Schedule   CLIMATE_anlaytics
2019-05-21 06:40:00.069   Action Group   CONTROLPAGE_CLIMATE_analytics
2019-05-21 06:40:00.086   Script   72.3 CP_TEMP_devon
2019-05-21 06:40:00.094   Script   70.0 CP_TEMP_hall2
2019-05-21 06:40:00.098   Script   71.7 CP_TEMP_hall3
2019-05-21 06:40:00.099   Script   72.7 CP_TEMP_kitchen
2019-05-21 06:40:00.103   Script   70.4 CP_TEMP_livingroom
2019-05-21 06:40:00.104   Script   70.1 CP_TEMP_mbr
2019-05-21 06:40:00.105   Script   74.7 CP_TEMP_den
2019-05-21 06:40:00.110   Script   72.8 CP_TEMP_thea
2019-05-21 06:40:00.111   Script   70.9 CP_TEMP_masterbath
2019-05-21 06:40:00.112   Script   70.1 CP_TEMP_mbr
2019-05-21 06:41:36.904   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall3" off
2019-05-21 06:43:00.062   Script   House Mode: dayTime
2019-05-21 06:48:21.622   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall2" on
2019-05-21 06:48:48.454   Timed Devices   "TIMEDDEVICE_ACTIVITY_masterbath" on
2019-05-21 06:49:02.743   Timed Devices   "TIMEDDEVICES_ACTIVITY_hall3" on
2019-05-21 06:49:54.966   Timed Devices   "TIMEDDEVICES_ACTIVITY_devon" on
2019-05-21 06:53:12.952   Timed Devices   "TIMEDDEVICES_ACTIVITY_livingroom" on
2019-05-21 06:54:00.050   Script   House Mode: dayTime
2019-05-21 06:57:29.978   Lutron RRA2/Caséta Debug   Received: Dimmer Thea Bedroom - Main Lights (4) level set to 100.0
2019-05-21 06:58:33.075   utilities   getPSEF error in  Line '904' ;  error='[Errno 23] Too many open files in system'
2019-05-21 06:58:33.078   Group Trigger Error   Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

2019-05-21 06:58:33.078   SQL Logger Error   exception trying to insert row into table variable_history_620210550 ("CPU_usage_pushover")
2019-05-21 06:58:33.079   SQL Logger Error   unable to open database file
2019-05-21 06:58:33.079   SQL Logger Error   INSERT INTO variable_history_620210550 (value) VALUES (%s);
2019-05-21 06:58:33.080   Group Trigger Error   Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

2019-05-21 06:58:33.081   SQL Logger Error   exception trying to insert row into table variable_history_689937002 ("CPU_usage_nanoleafcontroller")
2019-05-21 06:58:33.081   SQL Logger Error   unable to open database file
2019-05-21 06:58:33.081   SQL Logger Error   INSERT INTO variable_history_689937002 (value) VALUES (%s);
2019-05-21 06:58:33.082   Group Trigger Error   Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

2019-05-21 06:58:33.082   SQL Logger Error   exception trying to insert row into table variable_history_1018398405 ("CPU_usage_securityspyimage")
2019-05-21 06:58:33.082   SQL Logger Error   unable to open database file
2019-05-21 06:58:33.082   SQL Logger Error   INSERT INTO variable_history_1018398405 (value) VALUES (%s);
2019-05-21 06:58:33.084   Group Trigger Error   Error in plugin execution ServerReplacedElem:

Traceback (most recent call last):
  File "plugin.py", line 373, in variableUpdated
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1055, in stack
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1033, in getouterframes
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1008, in getframeinfo
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 532, in findsource
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 483, in getmodule
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 467, in getabsfile
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 23] Too many open files in system

2019-05-21 06:58:33.084   SQL Logger Error   exception trying to insert row into table variable_history_1096030000 ("CPU_usage_masquerade")
2019-05-21 06:58:33.085   SQL Logger Error   unable to open database file
2019-05-21 06:58:33.085   SQL Logger Error   INSERT INTO variable_history_1096030000 (value) VALUES (%s);
2019-05-21 06:58:33.085   Group Trigger Error   Error in plugin execution ServerReplacedElem:



I forget- what’s the easiest way to restart the server remotely?

Nm that part- I am using an older Applescript in an action group with BetterEmail to reset the server:

Code: Select all

tell application "Indigo 7.3"
   quit
end tell

# Give the server some time to fully shut down
delay 10

tell application "Indigo 7.3"
   activate
end tell


We're super hi-tech around here.

What’s in your scripts?

It’s more likely to be one of your own scripts that’s running away, than one of the published plugins, so it would be useful to look at those.


Sent from my iPhone using Tapatalk Pro

What’s in your scripts?

It’s more likely to be one of your own scripts that’s running away, than one of the published plugins, so it would be useful to look at those.


Sent from my iPhone using Tapatalk Pro

Bummer- I have one group of scripts that handles dynamic rotating image paths (DRIP in the log file) for cameras. All the info is stored in one Indigo Variable that holds a dict. It uses no files. I’ll take a look though.

Thanks for the input howartp, any other thoughts are appreciated.

I am very suspicious this is the prime error, related to my script that updates an Indigo Variable for Dynamic Refreshing Image Paths.

Code: Select all

2019-05-21 03:45:32.843   WebServer   http://localhost:8176/variable_url_redirect/redirect?name=DRIP_CURRENTURL_den load error: [Errno 54] Connection reset by peer
2019-05-21 03:45:33.899   WebServer   disconnecting from Indigo server
2019-05-21 03:45:33.899   Application   WebServer client disconnected


[jay (support)]

I am very suspicious this is the prime error, related to my script that updates an Indigo Variable for Dynamic Refreshing Image Paths.

Code: Select all

2019-05-21 03:45:32.843   WebServer   http://localhost:8176/variable_url_redirect/redirect?name=DRIP_CURRENTURL_den load error: [Errno 54] Connection reset by peer
2019-05-21 03:45:33.899   WebServer   disconnecting from Indigo server
2019-05-21 03:45:33.899   Application   WebServer client disconnected


While that may be a symptom, the variable_url_redirect IWS plugin doesn't open any files, so it wouldn't be the root cause of the issue. What exactly is the script that you're using? If it opens files then you'll definitely want to make sure that it always closes regardless of any issues that it happens to encounter.

I am very suspicious this is the prime error, related to my script that updates an Indigo Variable for Dynamic Refreshing Image Paths.

Code: Select all

2019-05-21 03:45:32.843   WebServer   http://localhost:8176/variable_url_redirect/redirect?name=DRIP_CURRENTURL_den load error: [Errno 54] Connection reset by peer
2019-05-21 03:45:33.899   WebServer   disconnecting from Indigo server
2019-05-21 03:45:33.899   Application   WebServer client disconnected


While that may be a symptom, the variable_url_redirect IWS plugin doesn't open any files, so it wouldn't be the root cause of the issue. What exactly is the script that you're using? If it opens files then you'll definitely want to make sure that it always closes regardless of any issues that it happens to encounter.

My scripts for this use NO files, when I was cooking up the implementation, it seemed the ability to access a dict from an Indigo Variable was way more efficient than using a file.

The script updates the url for dynamic url plugin, which in turn points to a camera that provides a refreshing image (about every second)

Q1: Is it possible that difficulty opening an image (in this case from an wires ip camera) could cause this kind of error?

Q2: Is there a way to detect the WebServer is shutting down (short of scanning the log)? Thanks Jay