Indigo Server & tccd uses high CPU w external Python scripts

I recently moved from my 2018 Mac mini to a new M2, and everything seemed to be running fine.

I always had many embedded Python scripts that ran every 1 second, and since the Indigo logs always complained about them not finishing with 10 seconds, I decided to convert them all to external Python script files as recommended.

Now with these changes, Indigo Server seems to use a lot of CPU, and also causes the TCC daemon (tccd) to also use a lot. Together, I think it causes the entire system to be unstable, and most apps become unresponsive, requiring a reboot, which doesn't last very long. I even caught Indigo Server using 757% in the screenshot below!

Of course now I have disabled the scripts as I take steps to combine some of them into a single script which is helpful, but wondered if this is a known behavior of Indigo Server? I don't want to be afraid in the future to add more scripts, but this has me concerned.

I created a screen recording to show how I came to this conclusion. Basically, every external Python script which runs every 1 second, will increase tccd CPU usage by about 1.5%, which adds up pretty quickly when you have 20 or 30 scripts.

And in case you are wondering why I am not using plugins like TP-Link or TED, I have actually been running them for quite some time, but the plugins have been problematic, and getting developer support to fix things can be challenging or impossible. The Python scripts were (are?) reliable, do only what I need, and I am in full control of them.

Screen Recording;

https://www.youtube.com/watch?v=njHS19qHYdc

Screenshot of Indigo Server at 757%

Screenshot 2023-09-09 at 11.52.04 AM.png (195.63 KiB) Viewed 843 times

Starting up an external script has a lot of overhead, as it’s a new process. Doing multiple scripts every second is a really bad idea. You should have those scripts loop and do whatever they do without exiting. Put a sleep() call in the loop.

Then just start those scripts once when the Indigo server starts.


Sent from my iPhone using Tapatalk

[jay (support)]

Questions:

1) Do you run the same scripts every 1 second (vs running a stay-open script that runs forever)? The former is definitely a bad idea, particularly if you have a whole lot of them. If you have a lot of scripts that you start up once a second, then theoretically they should all complete in 1 second since you really don't want the same script to run multiple times at the same time.
2) If they are stay-open, do you have a sleep() call in your loop? If not, you need to have one as that allows other threads to run.

I think we need to understand more fully what the scripts do, what you are trying to accomplish, etc.

I am totally down to have a script start and just have a loop with sleep, but I am not sure how to do that.

I put one of my newly combined scripts below that checks power usage of all my current TP-Links every 1 second.

I found time.sleep(1) but not sure how best to create the loop, or where exactly to put the sleep.

Code: Select all

#!/usr/bin/env python
#
# TP-Link Wi-Fi Smart Plug Protocol Client
# For use with TP-Link HS-100 or HS-110
# 
# by Lubomir Stroetmann
# Copyright 2016 softScheck GmbH
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#
import socket
import indigo
import json

# Encryption and Decryption of TP-Link Smart Home Protocol
# XOR Autokey Cipher with starting key = 171
def encrypt(string):
   key = 171
   result = b"\0\0\0\0"
   for i in string:
      a = key ^ i
      key = a
      result += bytes([a])
   return result

def decrypt(string):
   key = 171
   result = b""
   for i in string:
      a = key ^ i
      key = i
      result += bytes([a])
   return result.decode()

# Set target IP, port and command to send
ip = '192.168.20.142'
port = 9999
cmd = '{"emeter":{"get_realtime":{}}}'
var = indigo.variables[473517221]           #  Change this to the actual ID of your variable

# Send command and receive reply
try:
    sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_tcp.connect((ip, port))
    sock_tcp.send(encrypt(cmd.encode()))
    data = sock_tcp.recv(2048)
    sock_tcp.close()
   
#   indigo.server.log("Sent:     ", cmd)
#   indigo.server.log("Received: ", decrypt(data[4:]))
   
    jsonData = json.loads(decrypt(data[4:]))
    var.value = str(int(jsonData["emeter"]["get_realtime"]["power"]))
    var.replaceOnServer()

except socket.error:
    indigo.server.error("Could not connect to host " + ip + ":" + str(port))
   
# Set target IP, port and command to send
ip = '192.168.20.143'
port = 9999
cmd = '{"emeter":{"get_realtime":{}}}'
var = indigo.variables[34705482]           #  Change this to the actual ID of your variable

# Send command and receive reply
try:
    sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_tcp.connect((ip, port))
    sock_tcp.send(encrypt(cmd.encode()))
    data = sock_tcp.recv(2048)
    sock_tcp.close()
   
#   indigo.server.log("Sent:     ", cmd)
#   indigo.server.log("Received: ", decrypt(data[4:]))
   
    jsonData = json.loads(decrypt(data[4:]))
    var.value = str(int(jsonData["emeter"]["get_realtime"]["power"]))
    var.replaceOnServer()

except socket.error:
    indigo.server.error("Could not connect to host " + ip + ":" + str(port))


# Set target IP, port and command to send
ip = '192.168.20.144'
port = 9999
cmd = '{"emeter":{"get_realtime":{}}}'
var = indigo.variables[1822033147]           #  Change this to the actual ID of your variable

# Send command and receive reply
try:
    sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_tcp.connect((ip, port))
    sock_tcp.send(encrypt(cmd.encode()))
    data = sock_tcp.recv(2048)
    sock_tcp.close()
   
#   indigo.server.log("Sent:     ", cmd)
#   indigo.server.log("Received: ", decrypt(data[4:]))
   
    jsonData = json.loads(decrypt(data[4:]))
    var.value = str(int(jsonData["emeter"]["get_realtime"]["power"]))
    var.replaceOnServer()

except socket.error:
    indigo.server.error("Could not connect to host " + ip + ":" + str(port))
   
# Set target IP, port and command to send
ip = '192.168.20.145'
port = 9999
cmd = '{"emeter":{"get_realtime":{}}}'
var = indigo.variables[1879796290]           #  Change this to the actual ID of your variable

# Send command and receive reply
try:
    sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_tcp.connect((ip, port))
    sock_tcp.send(encrypt(cmd.encode()))
    data = sock_tcp.recv(2048)
    sock_tcp.close()
   
#   indigo.server.log("Sent:     ", cmd)
#   indigo.server.log("Received: ", decrypt(data[4:]))
   
    jsonData = json.loads(decrypt(data[4:]))
    var.value = str(int(jsonData["emeter"]["get_realtime"]["power"]))
    var.replaceOnServer()

except socket.error:
    indigo.server.error("Could not connect to host " + ip + ":" + str(port))

# Set target IP, port and command to send
ip = '192.168.20.146'
port = 9999
cmd = '{"emeter":{"get_realtime":{}}}'
var = indigo.variables[1716765030]           #  Change this to the actual ID of your variable

# Send command and receive reply
try:
    sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_tcp.connect((ip, port))
    sock_tcp.send(encrypt(cmd.encode()))
    data = sock_tcp.recv(2048)
    sock_tcp.close()
   
#   indigo.server.log("Sent:     ", cmd)
#   indigo.server.log("Received: ", decrypt(data[4:]))
   
    jsonData = json.loads(decrypt(data[4:]))
    var.value = str(int(jsonData["emeter"]["get_realtime"]["power"]))
    var.replaceOnServer()

except socket.error:
    indigo.server.error("Could not connect to host " + ip + ":" + str(port))

# Set target IP, port and command to send
ip = '192.168.20.147'
port = 9999
cmd = '{"emeter":{"get_realtime":{}}}'
var = indigo.variables[395059116]           #  Change this to the actual ID of your variable

# Send command and receive reply
try:
    sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_tcp.connect((ip, port))
    sock_tcp.send(encrypt(cmd.encode()))
    data = sock_tcp.recv(2048)
    sock_tcp.close()
   
#   indigo.server.log("Sent:     ", cmd)
#   indigo.server.log("Received: ", decrypt(data[4:]))
   
    jsonData = json.loads(decrypt(data[4:]))
    var.value = str(int(jsonData["emeter"]["get_realtime"]["power"]))
    var.replaceOnServer()

except socket.error:
    indigo.server.error("Could not connect to host " + ip + ":" + str(port))


# Set target IP, port and command to send
ip = '192.168.20.148'
port = 9999
cmd = '{"emeter":{"get_realtime":{}}}'
var = indigo.variables[831947697]           #  Change this to the actual ID of your variable

# Send command and receive reply
try:
    sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_tcp.connect((ip, port))
    sock_tcp.send(encrypt(cmd.encode()))
    data = sock_tcp.recv(2048)
    sock_tcp.close()
   
#   indigo.server.log("Sent:     ", cmd)
#   indigo.server.log("Received: ", decrypt(data[4:]))
   
    jsonData = json.loads(decrypt(data[4:]))
    var.value = str(int(jsonData["emeter"]["get_realtime"]["power"]))
    var.replaceOnServer()

except socket.error:
    indigo.server.error("Could not connect to host " + ip + ":" + str(port))

[jay (support)]

Ok, so not only are you potentially running lots of script instances, but you are also opening/closing a ton of socket connections for each of those. All those are HUGE RESOURCE SUCKS. And if any of those devices are slow to respond, then yet another socket connection will open to the device so you'll have multiple open socket connections. This is bad.

So, my first question has got to be why you aren't using the TP-Link (Kasa) plugin. That will be significantly better at resource management. What you are doing is really much better suited to a plugin.

However, if you insist on using a script, here's the framework for how you would need to rearchitect your solution. This would be an external script that you would load on server startup and never again - it would 1) open all socket connections, 2) loop reading/writing as necessary, 3) close down when the server wants you to.

Code: Select all

# Start by opening your sockets here - they will remain open during the running of your script
# ...
# Next, start your loop
while True:
    # Inside a try block, do your reading and writing as necessary
    # If there's an exception, log it but continue
    # ...
    # The last thing is to sleep for a second. Using this call will ensure that the server
    # can effectively shut down the script process and it'll give you the opportunity
    # to close the socket
    try:
        indigo.activePlugin.sleep(1)
    except:
        # If the server wants you to shut down, it will raise an exception from the sleep
        # method call. So here you'll want to close the socket connections.
        # ...
        break

Again, however, I have to say again that integrations like this are exactly why we built the plugin mechanism - it's just much better at helping you manage resources and protecting against the exact resource drains that your current solution are causing on your Mac.

BTW, I can totally see why the server could go up to 700% utilization - one misbehaving kasa device (not responding well or at all) will hang up your socket connection, then another iteration of the script would open another socket and would never receive a reply, etc, etc. All of those things could cause a potentially HUGE number of IndigoPluginHost processes (which also run external scripts), causing the server to get completely bogged down trying to manage potentially hundreds of processes. You have no timeouts on your socket connections, so by default they will timeout to about 30 seconds. So you could have 30 instances for just a single run before they start timing out. Multiply that by the number of different scripts you have and it's just a nightmare.

tccd becomes involved because your scripts are opening socket connections, which are protected resources.

Everything you said makes sense.

I did try using the Kasa plugin for many months, and although the developer did try to fix many of the issues I encountered, unfortunately the plugin was not usable for me. It would often lose communication with the devices but the plugin still thought the device was online. And I could show the devices never actually went offline, so I would just have days of missing data.

So it was back to Python scripts for me, which I used for years before the Kasa plugin, and never had any issues.

As of right now, I have the script listed above running every 1 second in an external Python file, and CPU usage is very reasonable. I will see how this works in the long term.

In the meantime, I am going to work on trying to rearchitect the script based on your suggestions, and have it load up once at startup. If I can get that working, I can apply this to other device scripts.

Thank You

Also, I would question the utility of a one second refresh for energy monitoring. That really seems like overkill.


Sent from my iPhone using Tapatalk

Also, I would question the utility of a one second refresh for energy monitoring. That really seems like overkill.

It’s a fair question, but I am often using this type of near real-time data on my phone to see what the energy usage changes are based on turning something on or off, so I like to see the change as immediate as possible.

I think the native Kasa app and TED web interface for example use 1 and 2 second refreshes respectively.