Indigo Domotics

A comprehensive "rule of thumb" for optimized use of Python scripts within Indigo.

Embedded Python Scripts
- will be executed within the existing Indigo Server Host Python process (see below.)
- will be terminated automatically if they do not complete within 10 seconds.
- embedded scripts will be executed consecutively; the second script will not begin until the first has finished (or has been terminated.)
- generally, it is most efficient to run scripts that will complete in under 10 seconds as embedded scripts.
- all embedded scripts should include an escape; otherwise, a hung script will take 10 seconds to "complete."
- there is only one process to run all embedded scripts.

Once the embedded script process is started it is left running indefinitely (waiting for the next embedded script to be sent to it from the Indigo Server). The result is that embedded scripts can start very quickly with little overhead (assuming an embedded script isn't already running in which case they are blocked until it is completed or the 10 second timeout)

Linked Python Scripts
- each linked script will be executed in its own (separate) process.
- will not be terminated automatically.
- scripts that take more than 10 seconds to complete should always be run as linked scripts.
- linked scripts will be executed concurrently (there is an OS limitation on the number of simultaneous processes can run at any one time; it's unlikely that users will run into this limitation.)
- all linked scripts should include an escape; otherwise, a hung script may never terminate.
- combining linked scripts to reduce the number of spawned processes may not significantly affect resources used.

General Tips
- linked scripts always start up a new process, there is some additional overhead to get them started. The process must be created and establish communication with the Indigo Server. It is relatively fast, but it is not recommended to start a hundred consecutive linked scripts that all do something simple (like changing an Indigo variable value). In that case an embedded script would be much more efficient for the CPU.
- there is no need to 'import indigo' into a script as indigo is automatically imported as a part of the host process.
- it may be helpful to schedule scripts to run at "odd" times to avoid conflicts.

Please provide any suggestions for additions, clarifications, or corrections.
Dave

DaveL17 wrote:

(there is a limitation of how many simultaneous processes can run at any one time; processes will be queued automatically until they can be run.)

Well, there's an OS limit to the number of processes, but it's pretty unlikely that that limit will be hit. I don't believe Indigo imposes any other limit to external processes.

DaveL17 wrote:

- generally, linked scripts should be combined as much as possible to reduce the number of spawned processes.

I think that's more a matter of opinion. Unless your Mac is significantly resource constrained it's really not an issue. I definitely wouldn't spend any time optimizing just to reduce the number of external scripts.

jay (support) wrote:

Well, there's an OS limit to the number of processes, but it's pretty unlikely that that limit will be hit. I don't believe Indigo imposes any other limit to external processes.

jay (support) wrote:

I think that's more a matter of opinion. Unless your Mac is significantly resource constrained it's really not an issue. I definitely wouldn't spend any time optimizing just to reduce the number of external scripts.

Thanks Jay - I will amend my post to include your points. My inference was based on the fact that I have a schedule that runs 8 linked scripts every 15 minutes. When this schedule runs, I see something akin to the following in the Indigo schedule window each time the schedule runs:


These delayed actions run within a matter of 15-30 seconds. Most of the scripts make a subprocess call to run an external Gnuplot script. Perhaps it's a system limitation on the number of simultaneous subprocess calls?
Dave

Those "_delayed_action_" prefixed items are just actions that in the UI have the optional delay checkbox enabled, and aren't related directly to how linked scripts are executed.

Note that embedded python scripts (but NOT AppleScripts) actually do run in their own process outside of the Indigo Server. But there is only 1 process to run all embedded python scripts so the result is they do not execute concurrently. Indigo Server manages killing and restarting that process if any embedded scripts take longer than 10 seconds to complete. So the behaviors you describe for embedded are correct, and it is just more of a technical detail that they actually run in their own process.

Once the embedded script process is started it is left running indefinitely (waiting for the next embedded script to be sent to it from the Indigo Server). The result is that embedded scripts can start very quickly with little overhead (assuming an embedded script isn't already running in which case they are blocked until it is completed or the 10 second timeout). On the other hand because linked scripts always start up a new process there is some additional overhead to get them started. The process has to be created and it has to establish communication with the Indigo Server. It is relatively fast, but I wouldn't recommend starting a hundred consecutive linked scripts that all do something simple (like changing and Indigo variable value). In that case an embedded script would be much more efficient for the CPU.

And thanks for putting together this list! I'm going to flag it to show at the top of this sub-forum.

matt (support) wrote:

Those "_delayed_action_" prefixed items are just actions that in the UI have the optional delay checkbox enabled, and aren't related directly to how linked scripts are executed.

Ha! What do you know about that? I forgot that I had put those scripts on a delay. I'm sure that I had a very good reason to do that at the time.

matt (support) wrote:

Note that embedded python scripts (but NOT AppleScripts) actually do run in their own process outside of the Indigo Server. But there is only 1 process to run all embedded python scripts so the result is they do not execute concurrently. Indigo Server manages killing and restarting that process if any embedded scripts take longer than 10 seconds to complete. So the behaviors you describe for embedded are correct, and it is just more of a technical detail that they actually run in their own process.

Once the embedded script process is started it is left running indefinitely (waiting for the next embedded script to be sent to it from the Indigo Server). The result is that embedded scripts can start very quickly with little overhead (assuming an embedded script isn't already running in which case they are blocked until it is completed or the 10 second timeout). On the other hand because linked scripts always start up a new process there is some additional overhead to get them started. The process has to be created and it has to establish communication with the Indigo Server. It is relatively fast, but I wouldn't recommend starting a hundred consecutive linked scripts that all do something simple (like changing and Indigo variable value). In that case an embedded script would be much more efficient for the CPU.

Thanks Matt. I'll add further clarity based on the above.

A side note for what it's worth... I tend to try not to run things on typical "round" periods of time. 1 hour, 30 minutes, 15 minutes 5 minutes ect... I usually try to have things run at odd times so I don't end up with the long queuing that you may be experiencing. Just an attempt to keep the highway as open as possible at all times

Dewster35 wrote:

A side note for what it's worth... I tend to try not to run things on typical "round" periods of time. 1 hour, 30 minutes, 15 minutes 5 minutes ect... I usually try to have things run at odd times so I don't end up with the long queuing that you may be experiencing. Just an attempt to keep the highway as open as possible at all times

Turns out these delays were absolutely of my own creation. But thanks for the pointer; it's a good tip!

How do I go about creating a "linked python script"?

mediabox wrote:

How do I go about creating a "linked python script"?

Create a new Trigger (or Schedule, or Action) and then on the Actions tab:

select Server Actions
select Script and File Actions
select Execute Script

Then, rather than pasting your script into the script window (Embedded), select File and point to a file where your script resides (linked).

Note that this must be done from the server machine and not another computer running the Indigo client software.

Dave

- all embedded scripts should include an escape; otherwise, a hung script will take 10 seconds to "complete."

When I search online for how to do this, all I get is how to "escape characters" which I know is not what I want.

Could someone provide an example please?

Different Computers wrote:

- all embedded scripts should include an escape; otherwise, a hung script will take 10 seconds to "complete."

When I search online for how to do this, all I get is how to "escape characters" which I know is not what I want.

Dave's referring to a design principle, not a specific function/method. He means that an embedded script should never hang up waiting for something to happen without some way to time it out. For instance, if the script issues a network call, it should set a timeout so that it won't wait forever for a reply but will rather timeout, giving the script a chance to shutdown before Indigo has to kill it.

Indigo forcing a script to quit is not ideal and can have unexpected side effects. Using the network example from above, if the script opens a network connect but doesn't set a short timeout, then potentially (depending on how the network connection was created) the network stack will retain that connection until it's default timeout (sometimes 30 seconds, sometimes even longer). Because the network stack will retain that open connection, it could be that any further attempts to create an identical connection will fail because it's still open.

Bottom line is that a script should finish it's work as quickly as possible so that it has a chance to do any cleanup that might be done at the end (closing a network connection, closing a file, etc.)

jay (support) wrote:

Dave's referring to a design principle, not a specific function/method.

Funny, that doesn't sound like me. In all seriousness though, IIRC, I pulled these tips together by scouring the forums for pointers--in addition to my own. I just don't want to take undue credit for someone else's point.

Or maybe it was me.

DaveL17 wrote:

Then, rather than pasting your script into the script window (Embedded), select File and point to a file where your script resides (linked).

Sorry... first time trying a linked script and I'm missing something. This is my script (runs fine as Embedded):

Code: Select all

import requests
url = 'http://IPADDRESS:Port/cgi-bin/CGIProxy.fcgi?cmd=snapPicture2&usr=user&pwd=password'
reply = requests.get(url)
reply.raise_for_status()
with open('/Users/williammoore/Desktop/IPCamRotate/Test.jpg', 'wb') as image_file:
    image_file.write(reply.content)

Then I saved on my desktop on the indigo server machine. I've tried saving as a txt, py, pyc... and when I run the action with the linked script... it opens it on the machine in textWrangler. I'm guessing either I need a different extension for the file or there is an introductory piece of code I need to tell it to run "additional overhead"?

I tried adding

Code: Select all

#!/usr/bin/indigohostapp/Contents/MacOS/IndigoPluginHost -x

to the top of the script (and a few other things from scouring the forum)... still opened in text wrangler.

Last question... what are some escape options I could add to my script to either quit running or try something else?

whmoorejr wrote:

Then I saved on my desktop on the indigo server machine. I've tried saving as a txt, py, pyc... and when I run the action with the linked script... it opens it on the machine in textWrangler. I'm guessing either I need a different extension for the file or there is an introductory piece of code I need to tell it to run "additional overhead"?

It should end in .py

Show a screen grab of the action you're configuring to run the script.

I'm trying to keep it all simple while I figure this stuff out.

Do I need to R-Click on the .py file and tell it to open with something else?