Plugin behaviour and CPU usage (after upgrading to 2022.2?)

Posted on
Wed Sep 13, 2023 1:07 pm
lochnesz offline
Posts: 370
Joined: Oct 01, 2014
Location: Stockholm, Sweden

Plugin behaviour and CPU usage (after upgrading to 2022.2?)

I had some strange issues with my Indigo server, possibly started after upgrading to 2022.2 but it could be a coincidence. Since then I tried to reinstall MacOS and restore Indigo from a backup (to start somewhat fresh again) but that didn't help.

So one of the issues is that the system seems to get slower over time. I notice it mainly though response time when I use DomoPad and turn on or off devices, or responsiveness of DomoPad control pages on my tablets. Also my DSC plugin fails to connect to my Envisalink EVL-4 very often, and the DSC plugin stops and then restarts, and this goes on. There can be periods when all works fine for a while, then it starts to happen again. I have tried to reboot the EVL-4 (many times) since I know that it can only have one connection and to avoid that a hanged session/connection causes the issue. All this worked very well before upgrading to 2022.2, but it could be a coincidence.

Searching for clues, I looked at the Activity Monitor and noticed that plugin CPU consumption rises quite much over time for one of the IndigoPluginHost3 processes (that's why I post in this forum and not the DSC (this time) or DomoPad forum). After a restart of the Indigo server, the highest CPU for one of the IndigoPluginHost3 processes is around 20-30%, but after a few days it is up to 80-90%. Is this normal?

See attached screenshots, the lower CPU consumption is directly after a Indigo server start, and the other with higher CPU consumption is after some days.

The error message from DSC plugin is:
Code: Select all
   DSC Alarm                       Initializing communication at address: 192.168.xxx.xxx:4025
   DSC Alarm                       Communication established
   DSC Alarm Error                 Connection RX Error: read failed: [Errno 54] Connection reset by peer
   DSC Alarm Error                 Connection TX Error: write failed: [Errno 32] Broken pipe
   Error                           plugin "DSC Alarm 2022.0.6" (pid 69047) unexpectedly stopped -- restarting in 5 seconds

   DSC Alarm                       Initializing communication at address: 192.168.xxx.xxx:4025
   DSC Alarm                       Communication established
   DSC Alarm Error                 Connection RX Error: read failed: [Errno 54] Connection reset by peer
   DSC Alarm Error                 Connection TX Error: write failed: [Errno 32] Broken pipe
   Error                           plugin "DSC Alarm 2022.0.6" (pid 69158) unexpectedly stopped -- restarting in 20 seconds

   Reloading plugin "DSC Alarm 2022.0.6" using API v3.0
   Starting plugin "DSC Alarm 2022.0.6" (pid 69164)
   Started plugin "DSC Alarm 2022.0.6"
   DSC Alarm                       
                                   ==================== Initializing New Plugin Session ======================================================
                                   Plugin name:         DSC Alarm
                                   Plugin version:      2022.0.6
                                   Plugin ID:           com.frightideas.indigoplugin.dscAlarm
                                   Plugin Log Level:    Info
                                   Indigo version:      2022.2.2
                                   Python version:      3.10.2 (v3.10.2:a58ebcc701, Jan 13 2022, 14:50:16) [Clang 13.0.0 (clang-1300.0.29.30)]
                                   Mac OS Version:      12.6.8
                                   Process ID:          69164
                                   ===========================================================================================================

   DSC Alarm                       Initializing communication at address: 192.168.xxx.xxx:4025
   DSC Alarm Error                 Error opening socket: Could not open port socket://192.168.xxx.xxx:4025: timed out
   DSC Alarm Warning               Plugin will attempt to re-initialize again in 3 minutes.


The pid is not the one consuming much CPU and is only active for a short time.

When I reboot the Mac AND the EVL4 at the same time it usually works for a while...

I don't know if these thing are in any way related, but there is something strange going on. Could it be that something happened with the 2022.2 upgrade, some process still running from the previous version (even though I reinstalled MacOS)?
How would I go forward finding the issue?

If this is posted in the wrong forum, go ahead and move it please.
Attachments
Screenshot 2023-09-13 at 20.16.14.png
Screenshot 2023-09-13 at 20.16.14.png (284.71 KiB) Viewed 1302 times
Screenshot 2023-09-13 at 20.23.28.png
Screenshot 2023-09-13 at 20.23.28.png (272.59 KiB) Viewed 1302 times

Posted on
Wed Sep 13, 2023 2:57 pm
jay (support) offline
Site Admin
User avatar
Posts: 18224
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

When it happens again, select the IPH process that has high CPU utilization, then click the "i" button above the list. In the resulting window, switch to the Open Files and Ports tab, and the first line or two should show you which plugin it is. Disable that one and see if you see any further issues - my guess is that you have a plugin that's getting off into the weeds...

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Sep 14, 2023 12:05 am
lochnesz offline
Posts: 370
Joined: Oct 01, 2014
Location: Stockholm, Sweden

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

Here is what shows in the Open Files and Ports tab for that process (up to around 45% CPU load now):

Code: Select all
cwd
/Library/Application Support/Perceptive Automation/Indigo 2022.2/Plugins/.Web Server.indigoPlugin/Contents/Server Plugin
txt
/Library/Application Support/Perceptive Automation/Indigo 2022.2/IndigoPluginHost3.app/Contents/MacOS/IndigoPluginHost3
txt
/Library/Preferences/Logging/.plist-cache.A9HwMra5
txt
/private/var/db/timezone/tz/2023c.1.0/icutz/icutz44l.dat
txt
/private/var/folders/4g/mwvqnf0n6_l6358y8l_bwbt80000gn/0/com.apple.LaunchServices.dv/com.apple.LaunchServices-3028-v2.csstore
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/SystemAppearance.car
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/math.cpython-310-darwin.so
txt
/System/Library/TextEncodings/Unicode Encodings.bundle/Contents/MacOS/Unicode Encodings
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_socket.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_heapq.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_json.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/Python
txt
/usr/share/icu/icudt70l.dat
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/Aqua.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/FauxVibrantLight.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLight.car
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_datetime.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/array.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_struct.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/select.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_elementtree.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_queue.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/readline.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_bisect.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_pickle.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/libncursesw.5.dylib
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/fcntl.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/pyexpat.cpython-310-darwin.so
txt
/Library/Application Support/Perceptive Automation/Indigo 2022.2/Plugins/.Web Server.indigoPlugin/Contents/Server Plugin/plugin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_bz2.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/termios.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_opcode.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_hashlib.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_blake2.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/binascii.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_random.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_sha512.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_uuid.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/.dylibs/libgcc_s.1.1.dylib
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_contextvars.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/core/_multiarray_tests.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/.dylibs/libquadmath.0.dylib
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/websockets/speedups.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/libssl.1.1.dylib
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/fft/_pocketfft_internal.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_posixsubprocess.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/linalg/_umath_linalg.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/_philox.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/_mt19937.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/bit_generator.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/_sfc64.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/_pcg64.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/_common.cpython-310-darwin.so
txt
/Library/Application Support/Perceptive Automation/Indigo 2022.2/Plugins/.Web Server.indigoPlugin/Contents/Server Plugin/httpauth.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/multidict/_multidict.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_scproxy.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_asyncio.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/zlib.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/markupsafe/_speedups.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/libcrypto.1.1.dylib
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/mtrand.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/_bounded_integers.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_ssl.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httptools/parser/url_parser.cpython-310-darwin.so
txt
/Library/Application Support/Perceptive Automation/Indigo 2022.2/Plugins/.Web Server.indigoPlugin/Contents/Server Plugin/ssl_utils.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httptools/parser/parser.cpython-310-darwin.so
txt
/Library/Application Support/Perceptive Automation/Indigo 2022.2/Plugins/.Web Server.indigoPlugin/Contents/Server Plugin/file_lookup.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ujson.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/random/_generator.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_lzma.cpython-310-darwin.so
txt
/private/var/db/analyticsd/events.allowlist
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/uvloop/loop.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_decimal.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/.dylibs/libgfortran.5.dylib
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
txt
/usr/lib/dyld
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/unicodedata.cpython-310-darwin.so
txt
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
0
/dev/null
1
/dev/null
2
/dev/null
3
localhost:49158->localhost:indigo-server
4
/Library/Application Support/Perceptive Automation/Indigo 2022.2/Logs/indigoplugin.webserver/plugin.log
5
->0xd80f0c8d2162571c
6
->0x3d10554162ebaccd
7
count=1, state=0x8
8
->0x7b48aec1d1b15f38
9
->0x710c2eae405ce2f4
10
->0xf19c897594a83672
11
->0x47048de98d8fe0d0
12
->0x2288e7cdc2eb644b
13
->0x3e4306dbc8c9f783
14
*:8176
15
/dev/null
16
admins-mac-mini.localdomain:8176->tab-4.localdomain:49102
17
admins-mac-mini.localdomain:8176->tab-1.localdomain:38882
18
admins-mac-mini.localdomain:8176->tab-1.localdomain:38884
19
admins-mac-mini.localdomain:8176->tab-4.localdomain:49098
20
admins-mac-mini.localdomain:8176->tab-1.localdomain:38890
21
admins-mac-mini.localdomain:8176->tab-1.localdomain:35150
22
admins-mac-mini.localdomain:8176->android-2103b371b7e60dd9.localdomain:42377
23
admins-mac-mini.localdomain:8176->tab-1.localdomain:38886
24
admins-mac-mini.localdomain:8176->tab-4.localdomain:57452
25
admins-mac-mini.localdomain:8176->tab-1.localdomain:38892
26
admins-mac-mini.localdomain:8176->tab-4.localdomain:49106
27
admins-mac-mini.localdomain:8176->tab-4.localdomain:49108
28
admins-mac-mini.localdomain:8176->gw:51525
29
admins-mac-mini.localdomain:8176->tab-4.localdomain:49094
30
admins-mac-mini.localdomain:8176->tab-1.localdomain:38878
31
admins-mac-mini.localdomain:8176->tab-1.localdomain:38888
32
admins-mac-mini.localdomain:8176->android-2103b371b7e60dd9.localdomain:42370
33
admins-mac-mini.localdomain:8176->tab-4.localdomain:49104
34
admins-mac-mini.localdomain:8176->tab-1.localdomain:38880
35
admins-mac-mini.localdomain:8176->tab-4.localdomain:49096
36
admins-mac-mini.localdomain:60347->carport.localdomain:http
37
admins-mac-mini.localdomain:8176->tab-4.localdomain:49100
38
admins-mac-mini.localdomain:60348->groventre.localdomain:http
123
admins-mac-mini.localdomain:8176->android-d8c9182691d80495.localdomain:35276
131
admins-mac-mini.localdomain:8176->gw:51524

Posted on
Thu Sep 14, 2023 10:06 am
jay (support) offline
Site Admin
User avatar
Posts: 18224
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

Interesting, that's the Indigo Web Server. Some questions:

1) It appears that you have at least 2 web browsers loading the Indigo Touch Web client (ITW), but there are also some other things as well (a couple of Android entries, a couple of connections from something named gw, a connection from something named carport, a connection from something named groventre). What are those other things and what are they doing?
2) Do you have something (variables, devices, etc) that update very frequently?

We don't know which (if any) of those connections are very busy based on the Activity Monitor information, so we'll need you to help characterize what those connections might be doing.

Also, what Mac are you using and what macOS version?

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Fri Sep 15, 2023 12:39 am
lochnesz offline
Posts: 370
Joined: Oct 01, 2014
Location: Stockholm, Sweden

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

It is a Dual-Core i5 2,6 GHz Mac Mini Late 2014 with 16 GB RAM.

There are three variables updating every second, two time/date inserts in variables in different formats and a variable updating (toggled) every second which is used to verify that the server is running.

There are 4-5 Android devices presenting control pages with DomoPad, and a iPad presenting a control page with IndigoTouch. On the control pages there are some camera images refreshing every second. The devices you mention "groventre" and "carport" are some of the cameras presented on the control pages. Some camera images are requested directly from the cameras and some are requested from BlueIris. The device gw is my Unifi USG, I don't know how that is having a connection to the IWS.

Posted on
Fri Sep 15, 2023 12:49 pm
matt (support) offline
Site Admin
User avatar
Posts: 21417
Joined: Jan 27, 2003
Location: Texas

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

To narrow down the problem I suggest you temporarily disable and close everything you listed to see what brings down the CPU utilization the most.

Also take a look at the memory usage over time of IndigoServer and that IndigoPluginHost process to see if it is creepy higher and higher (over days).

Image

Posted on
Sat Sep 23, 2023 7:45 am
lochnesz offline
Posts: 370
Joined: Oct 01, 2014
Location: Stockholm, Sweden

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

So the items that drive the CPU load are the camera images on the control pages. Each camera image with a refresh rate of 1 sec adds around 10% CPU. Continuous refresh adds about the double. This is with a control page on a iPad running Indigo Touch. I don't remember this ever being an issue before. Cameras are Ubiquity Dome G3 1080.

Posted on
Sat Sep 23, 2023 9:35 am
matt (support) offline
Site Admin
User avatar
Posts: 21417
Joined: Jan 27, 2003
Location: Texas

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

We'll look into this. If you view one of the camera images in a browser and then save it, how many MB is it?

Image

Posted on
Sat Sep 23, 2023 1:45 pm
lochnesz offline
Posts: 370
Joined: Oct 01, 2014
Location: Stockholm, Sweden

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

Around 150-200 KB for the ones with higher resolution direct from a camera, and around 20 KB for some smaller images resized by Blue Iris. One control page can have 2-4 images.

Posted on
Tue Oct 03, 2023 3:52 pm
matt (support) offline
Site Admin
User avatar
Posts: 21417
Joined: Jan 27, 2003
Location: Texas

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

The recent Indigo 2023.1.0 release should address this CPU utilization problem (related to refreshing images on Control Pages).

Image

Posted on
Wed Oct 04, 2023 6:57 am
lochnesz offline
Posts: 370
Joined: Oct 01, 2014
Location: Stockholm, Sweden

Re: Plugin behaviour and CPU usage (after upgrading to 2022.

Great, I have installed the upgrade now.

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 12 guests