Python execution timeout issue

[jay (support)]

Hmm, run each cam command in its own thread. I'll have to look into that. I took a peek at your link for that, Jay, a bit beyond my Python skills at this point, but I'll study some more. I think I ran into that, or something similar, when I was searching for the timeout solution, as others have had similar issues trying to speed up Python requests.get calls. I read things like "multiprocessing.dummy.Pool" and "pool.apply_async(requests.get, [params])" but stumbled on exactly how to implement those in my case (if those even apply to my case!).

Yeah, it's definitely not a beginner topic. I'll see if I can cook up an example.

Matt, you mentioned that Python scripts run parallel to other Indigo Server processing. Does that mean I could turn on a light even while my long-winded embedded python script was running? If so, then why the 10 second limit (and error) for python script execution?

The Indigo Server is threaded - that is, when possible, it does things as much as it can in parallel. Embedded scripts run in their own shared process, so every time the server needs to run one it just adds it to a queue that the shared process uses to execute scripts (one at a time). However, the server's other processes continue to run in parallel. So, the answer is yes, lights should continue to turn on and off, schedules run, etc. The only thing that's serial are any embedded scripts, which run one at a time in their own process and thus have the 10 second limitation. Why not just run them all in their own process? Because starting up a script execution process takes more time and a few more resources on your Mac. And, in fact, that's exactly what executing from a file does.

Anyway, it's working well enough for now. So I'll revisit this later.

By the way, it was a pleasant surprise that my .py files find each other (for importing) without any special installation, as long as they're in the same directory. And that they can find Indigo (to be able to import "indigo"), too (for logging). Is that because they're in the attachments folder? Or would a py script be able execute, and be able to import "indigo", no matter where it was located?

Python interpreters have something called a path, on which they search for modules that are imported. The first place they look is where the directory where the current script is running, so any other files in the same directory will be found automatically. Indigo uses a special wrapper environment, called the IndigoPluginHost, which runs all Indigo script. One of it's jobs is to add to the PYTHONPATH such that anything in the /Scripts/Attachments folder in the Indigo installation directory is on python's search path. Another thing IPH does is automatically add the indigo module so it's always available for you without an explicit import.

So many questions... when I modify a py script, what is the proper way to reintroduce it to Indigo? I can see that something compiles my script (creates a .pyc version). after it first executes. Is Indigo doing that, or the OS's python interpreter? Also, I was using the "Reload Libraries and Attachments" menu command after each edit, but when looking at the log, I'm not convinced that is doing anything for python attachments. So how best to "tell" Indigo I've modified a py attachment?

The python interpreter automatically creates a bytecode file (.pyc) from your script source whenever the script is run IF there is no .pyc file or if your .py file is newer. Reload Libraries and Attachments basically shuts down the IndigoPluginHost shared process that runs embedded scripts. Because it's a shared environment, if a script has already imported a module, it may not pick up any changes (because of how we wrap it). Also, and I'm not positive on this, but it may also not find any new files added to the path after it started up (though, again, I could be wrong on this one).

And thanks for all your support and walking me through this obscure issue...

No problem, it was a head scratcher.

[jay (support)]

Sorry, one more question. When I upgrade Indigo (like from 7.3 to 7.4), will the installer copy over these py files in the attachment folder for me?

Unfortunately, for now you still have to move over your script files from the old /Scripts folder to the new one. We had hoped to get that automated in 7.4 but ran out of time. Hopefully it will be automated in an upcoming release.

And will the 7.4 Action Groups find the copied external files, in the 7.4 folder, or will the Action Groups still be looking for the linked files in the 7.3 folder path? In other words, when upgrading, do I have to mind attachments, or does Indigo figure all that out? So if I modify a 7.4 attachment in the future, 7.4 will use that version, and not the 7.3 version that might still exist in the /Library/Application Support/Perceptive Automation/ directory? (I tend to leave previous installations in place until I know the new version is doing everything correctly and finding everything I modified in the previous Perceptive Automation folder.)

If you specify the file directly (executing the script as an file), then no, you'll need to change the action yourself. If you're just referencing them through an import statement in other scripts, then you don't have to change anything as Indigo sets up the PYTHONPATH for you (see my previous post) and it will point to the right Scripts folder.

The python interpreter automatically creates a bytecode file (.pyc) from your script source whenever the script is run IF there is no .pyc file or if your .py file is newer. Reload Libraries and Attachments basically shuts down the IndigoPluginHost shared process that runs embedded scripts. Because it's a shared environment, if a script has already imported a module, it may not pick up any changes (because of how we wrap it). Also, and I'm not positive on this, but it may also not find any new files added to the path after it started up (though, again, I could be wrong on this one).

While building and troubleshooting my py files, I thought I noticed that Indigo wasn't always using the latest version. So how do I make sure it does? After modifying a py file, I would trash the old pyc version, and use the Reload Libraries and Attachments command, too. Would that do the trick? Or is that overkill?

If you specify the file directly (executing the script as an file), then no, you'll need to change the action yourself. If you're just referencing them through an import statement in other scripts, then you don't have to change anything as Indigo sets up the PYTHONPATH for you (see my previous post) and it will point to the right Scripts folder.

OK, good to know...

[jay (support)]

While building and troubleshooting my py files, I thought I noticed that Indigo wasn't always using the latest version. So how do I make sure it does? After modifying a py file, I would trash the old pyc version, and use the Reload Libraries and Attachments command, too. Would that do the trick? Or is that overkill?

Yes, though you don't need to delete the pyc version, Python will do the work, you just have to use the reload menu item.

OK, thanks. I find myself falling down a familiar rabbit hole. With my first Indigo/Insteon system, in my previous home, I really got into it. Especially with AppleScript. I easily have four-figures of lines of code. I had my computers and Indigo doing some really complicated stuff. But then Indigo would update, or Apple would alter AppleScript support (iTunes!!!) and some component of my system would fall apart. And keeping track of what Indigo files I modified when upgrading... daunting.

I tried to bring it all over to the new home, but that was not 100% successful. So I bit the bullet (following your advice) and reset the modem and started my Indigo database from scratch. Considering my 90+ devices, and countless scripts and actions and triggers... not for the faint of heart. And after installing my old devices, and finding even worse reliability issues than in the old house, I took the hit and replaced all my old non-RF devices with new dual band devices. Not for the faint of wallet.

But this was the time to simplify, recreate only the capabilities I really needed, and, of course, to strip out all the AppleScript.

Virtually all said and done, my system is considerably faster and more robust, and waaaaay more reliable (dual band, yah!!). So now that it's over, I gotta admit it was all worth it, because my house is finally working as the HA fantasy has been promising all along.

But...

I now have to be careful that I don't re-complicate things (like with Action Groups calling py attachments calling other py attachments!!). It's very addictive, the challenge of getting it all to work the way I want it to...

One note of caution when working with external Python scripts. When naming your scripts, avoid using names that are used for standard Python modules. You'll get ImportErrors that aren't easy to interpret. For example, say you have a script to change dates into a format you prefer. If you call that file 'datetime.py' you'll get an error when you try to call it. Give them unique names and you'll be safe. This has bitten me more than once and will drive you crazy until you figure it out.

I shudder to think what would happen if you named your script 'indigo.py'. I think it would likely blow a hole in the space/time continuum.

Ha, good advice. So far I think I am still in the original dimension! Here's a sampling of my file names. I use "indigo" to remind me they belong to my HA system, then everything else in the name should keep them unique within Python. The names match (for the most part) the action groups that rely on them, so that also helps me stay organized...

indigo_cams_all_ir_auto.py
indigo_cams_all_ir_off.py
indigo_cams_all_preset_1.py
indigo_cams_all_preset_2.py
indigo_cams_all_reboot.py
indigo_cams.py
indigo_media.py

One thing (of many) that's a bit unexpected, my Attachments folder now includes a couple .pyc files, which I understand are the compiled versions of my scripts, but the interpreter has not created those for all my scripts, just two of the seven... Weird...

BTW, thanks to you, too, for your guidance...

[jay (support)]

One thing (of many) that's a bit unexpected, my Attachments folder now includes a couple .pyc files, which I understand are the compiled versions of my scripts, but the interpreter has not created those for all my scripts, just two of the seven... Weird...

The python interpreter dynamically compiles module files based on need. So unless it absolutely has to use something, it won't get compiled.