Optimizing Python Script Execution

Posted on
Sat Jan 02, 2016 9:36 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Optimizing Python Script Execution

A comprehensive "rule of thumb" for optimized use of Python scripts within Indigo.

Embedded Python Scripts
- will be executed within the existing Indigo Server Host Python process (see below.)
- will be terminated automatically if they do not complete within 10 seconds.
- embedded scripts will be executed consecutively; the second script will not begin until the first has finished (or has been terminated.)
- generally, it is most efficient to run scripts that will complete in under 10 seconds as embedded scripts.
- all embedded scripts should include an escape; otherwise, a hung script will take 10 seconds to "complete."
- there is only one process to run all embedded scripts.

Once the embedded script process is started it is left running indefinitely (waiting for the next embedded script to be sent to it from the Indigo Server). The result is that embedded scripts can start very quickly with little overhead (assuming an embedded script isn't already running in which case they are blocked until it is completed or the 10 second timeout)

Linked Python Scripts
- each linked script will be executed in its own (separate) process.
- will not be terminated automatically.
- scripts that take more than 10 seconds to complete should always be run as linked scripts.
- linked scripts will be executed concurrently (there is an OS limitation on the number of simultaneous processes can run at any one time; it's unlikely that users will run into this limitation.)
- all linked scripts should include an escape; otherwise, a hung script may never terminate.
- combining linked scripts to reduce the number of spawned processes may not significantly affect resources used.

General Tips
- linked scripts always start up a new process, there is some additional overhead to get them started. The process must be created and establish communication with the Indigo Server. It is relatively fast, but it is not recommended to start a hundred consecutive linked scripts that all do something simple (like changing an Indigo variable value). In that case an embedded script would be much more efficient for the CPU.
- there is no need to 'import indigo' into a script as indigo is automatically imported as a part of the host process.
- it may be helpful to schedule scripts to run at "odd" times to avoid conflicts.

Please provide any suggestions for additions, clarifications, or corrections.
Dave
Last edited by DaveL17 on Sat Jan 02, 2016 2:03 pm, edited 3 times in total.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Jan 02, 2016 10:44 am
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Optimizing Python Script Execution

DaveL17 wrote:
(there is a limitation of how many simultaneous processes can run at any one time; processes will be queued automatically until they can be run.)


Well, there's an OS limit to the number of processes, but it's pretty unlikely that that limit will be hit. I don't believe Indigo imposes any other limit to external processes.

DaveL17 wrote:
- generally, linked scripts should be combined as much as possible to reduce the number of spawned processes.


I think that's more a matter of opinion. Unless your Mac is significantly resource constrained it's really not an issue. I definitely wouldn't spend any time optimizing just to reduce the number of external scripts.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Sat Jan 02, 2016 11:09 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Optimizing Python Script Execution

jay (support) wrote:
Well, there's an OS limit to the number of processes, but it's pretty unlikely that that limit will be hit. I don't believe Indigo imposes any other limit to external processes.


jay (support) wrote:
I think that's more a matter of opinion. Unless your Mac is significantly resource constrained it's really not an issue. I definitely wouldn't spend any time optimizing just to reduce the number of external scripts.


Thanks Jay - I will amend my post to include your points. My inference was based on the fact that I have a schedule that runs 8 linked scripts every 15 minutes. When this schedule runs, I see something akin to the following in the Indigo schedule window each time the schedule runs:

Screen Shot 2016-01-02 at 11.00.05 AM.png
Screen Shot 2016-01-02 at 11.00.05 AM.png (24.36 KiB) Viewed 14277 times

These delayed actions run within a matter of 15-30 seconds. Most of the scripts make a subprocess call to run an external Gnuplot script. Perhaps it's a system limitation on the number of simultaneous subprocess calls?
Dave

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Jan 02, 2016 11:20 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Optimizing Python Script Execution

Those "_delayed_action_" prefixed items are just actions that in the UI have the optional delay checkbox enabled, and aren't related directly to how linked scripts are executed.

Note that embedded python scripts (but NOT AppleScripts) actually do run in their own process outside of the Indigo Server. But there is only 1 process to run all embedded python scripts so the result is they do not execute concurrently. Indigo Server manages killing and restarting that process if any embedded scripts take longer than 10 seconds to complete. So the behaviors you describe for embedded are correct, and it is just more of a technical detail that they actually run in their own process.

Once the embedded script process is started it is left running indefinitely (waiting for the next embedded script to be sent to it from the Indigo Server). The result is that embedded scripts can start very quickly with little overhead (assuming an embedded script isn't already running in which case they are blocked until it is completed or the 10 second timeout). On the other hand because linked scripts always start up a new process there is some additional overhead to get them started. The process has to be created and it has to establish communication with the Indigo Server. It is relatively fast, but I wouldn't recommend starting a hundred consecutive linked scripts that all do something simple (like changing and Indigo variable value). In that case an embedded script would be much more efficient for the CPU.

And thanks for putting together this list! I'm going to flag it to show at the top of this sub-forum.

Image

Posted on
Sat Jan 02, 2016 1:23 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Optimizing Python Script Execution

matt (support) wrote:
Those "_delayed_action_" prefixed items are just actions that in the UI have the optional delay checkbox enabled, and aren't related directly to how linked scripts are executed.


Ha! What do you know about that? I forgot that I had put those scripts on a delay. I'm sure that I had a very good reason to do that at the time. :)

matt (support) wrote:
Note that embedded python scripts (but NOT AppleScripts) actually do run in their own process outside of the Indigo Server. But there is only 1 process to run all embedded python scripts so the result is they do not execute concurrently. Indigo Server manages killing and restarting that process if any embedded scripts take longer than 10 seconds to complete. So the behaviors you describe for embedded are correct, and it is just more of a technical detail that they actually run in their own process.

Once the embedded script process is started it is left running indefinitely (waiting for the next embedded script to be sent to it from the Indigo Server). The result is that embedded scripts can start very quickly with little overhead (assuming an embedded script isn't already running in which case they are blocked until it is completed or the 10 second timeout). On the other hand because linked scripts always start up a new process there is some additional overhead to get them started. The process has to be created and it has to establish communication with the Indigo Server. It is relatively fast, but I wouldn't recommend starting a hundred consecutive linked scripts that all do something simple (like changing and Indigo variable value). In that case an embedded script would be much more efficient for the CPU.

Thanks Matt. I'll add further clarity based on the above.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Jan 02, 2016 1:50 pm
Dewster35 offline
Posts: 1030
Joined: Jul 06, 2010
Location: Petoskey, MI

Re: Optimizing Python Script Execution

A side note for what it's worth... I tend to try not to run things on typical "round" periods of time. 1 hour, 30 minutes, 15 minutes 5 minutes ect... I usually try to have things run at odd times so I don't end up with the long queuing that you may be experiencing. Just an attempt to keep the highway as open as possible at all times :)

Posted on
Sat Jan 02, 2016 2:01 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Optimizing Python Script Execution

Dewster35 wrote:
A side note for what it's worth... I tend to try not to run things on typical "round" periods of time. 1 hour, 30 minutes, 15 minutes 5 minutes ect... I usually try to have things run at odd times so I don't end up with the long queuing that you may be experiencing. Just an attempt to keep the highway as open as possible at all times :)

Turns out these delays were absolutely of my own creation. But thanks for the pointer; it's a good tip!

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Jul 02, 2016 8:16 pm
mediabox offline
Posts: 48
Joined: Apr 15, 2016
Location: chicago, il

Re: Optimizing Python Script Execution

How do I go about creating a "linked python script"?

Posted on
Sun Jul 03, 2016 4:01 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Optimizing Python Script Execution

mediabox wrote:
How do I go about creating a "linked python script"?

Create a new Trigger (or Schedule, or Action) and then on the Actions tab:

select Server Actions
select Script and File Actions
select Execute Script

Then, rather than pasting your script into the script window (Embedded), select File and point to a file where your script resides (linked).

Note that this must be done from the server machine and not another computer running the Indigo client software.

Dave

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri May 04, 2018 1:08 pm
Different Computers offline
User avatar
Posts: 2533
Joined: Jan 02, 2016
Location: East Coast

Re: Optimizing Python Script Execution

- all embedded scripts should include an escape; otherwise, a hung script will take 10 seconds to "complete."


When I search online for how to do this, all I get is how to "escape characters" which I know is not what I want.

Could someone provide an example please?

SmartThings refugee, so happy to be on Indigo. Monterey on a base M1 Mini w/Harmony Hub, Hue, DomoPad, Dynamic URL, Device Extensions, HomeKitLink, Grafana, Plex, uniFAP, Fantastic Weather, Nanoleaf, LED Simple Effects, Bond Home, Camect.

Posted on
Fri May 04, 2018 1:31 pm
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Optimizing Python Script Execution

Different Computers wrote:
- all embedded scripts should include an escape; otherwise, a hung script will take 10 seconds to "complete."


When I search online for how to do this, all I get is how to "escape characters" which I know is not what I want.


Dave's referring to a design principle, not a specific function/method. He means that an embedded script should never hang up waiting for something to happen without some way to time it out. For instance, if the script issues a network call, it should set a timeout so that it won't wait forever for a reply but will rather timeout, giving the script a chance to shutdown before Indigo has to kill it.

Indigo forcing a script to quit is not ideal and can have unexpected side effects. Using the network example from above, if the script opens a network connect but doesn't set a short timeout, then potentially (depending on how the network connection was created) the network stack will retain that connection until it's default timeout (sometimes 30 seconds, sometimes even longer). Because the network stack will retain that open connection, it could be that any further attempts to create an identical connection will fail because it's still open.

Bottom line is that a script should finish it's work as quickly as possible so that it has a chance to do any cleanup that might be done at the end (closing a network connection, closing a file, etc.)

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Fri May 04, 2018 8:52 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Optimizing Python Script Execution

jay (support) wrote:
Dave's referring to a design principle, not a specific function/method.

Funny, that doesn't sound like me. In all seriousness though, IIRC, I pulled these tips together by scouring the forums for pointers--in addition to my own. I just don't want to take undue credit for someone else's point.

Or maybe it was me.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue Jul 23, 2019 7:53 am
whmoorejr offline
User avatar
Posts: 762
Joined: Jan 15, 2013
Location: Houston, TX

Re: Optimizing Python Script Execution

DaveL17 wrote:
Then, rather than pasting your script into the script window (Embedded), select File and point to a file where your script resides (linked).


Sorry... first time trying a linked script and I'm missing something. This is my script (runs fine as Embedded):
Code: Select all
import requests
url = 'http://IPADDRESS:Port/cgi-bin/CGIProxy.fcgi?cmd=snapPicture2&usr=user&pwd=password'
reply = requests.get(url)
reply.raise_for_status()
with open('/Users/williammoore/Desktop/IPCamRotate/Test.jpg', 'wb') as image_file:
    image_file.write(reply.content)


Then I saved on my desktop on the indigo server machine. I've tried saving as a txt, py, pyc... and when I run the action with the linked script... it opens it on the machine in textWrangler. I'm guessing either I need a different extension for the file or there is an introductory piece of code I need to tell it to run "additional overhead"?

I tried adding
Code: Select all
#!/usr/bin/indigohostapp/Contents/MacOS/IndigoPluginHost -x
to the top of the script (and a few other things from scouring the forum)... still opened in text wrangler.

Last question... what are some escape options I could add to my script to either quit running or try something else?

Bill
My Plugin: My People

Posted on
Tue Jul 23, 2019 8:13 am
FlyingDiver offline
User avatar
Posts: 7189
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Optimizing Python Script Execution

whmoorejr wrote:
Then I saved on my desktop on the indigo server machine. I've tried saving as a txt, py, pyc... and when I run the action with the linked script... it opens it on the machine in textWrangler. I'm guessing either I need a different extension for the file or there is an introductory piece of code I need to tell it to run "additional overhead"?


It should end in .py

Show a screen grab of the action you're configuring to run the script.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Tue Jul 23, 2019 8:48 am
whmoorejr offline
User avatar
Posts: 762
Joined: Jan 15, 2013
Location: Houston, TX

Re: Optimizing Python Script Execution

I'm trying to keep it all simple while I figure this stuff out. :D

Do I need to R-Click on the .py file and tell it to open with something else?
Attachments
Python Linked Test.jpeg
Python Linked Test.jpeg (64.78 KiB) Viewed 9153 times

Bill
My Plugin: My People

Who is online

Users browsing this forum: No registered users and 4 guests

cron