Python as subprocess

I am updating a no longer maintained plugin to Python 3 and having some issue with a section that runs a python script as a subprocess.

I can run the code in a normal method, but then the whole of indigo hangs when it tries to run whilst I have an edit device window open, so I am back to trying to make it run as a subprocess.
I have it running but it errors on when trying to import PIL, however import os and sys works, so it's specific to that package.

The code I am running is

Code: Select all

   imgResult = subprocess.run(['python3', pypath+'text2png.py', params], stdout=outputInfo, stderr=errorInfo)


and it errors on this

Code: Select all

# Import the graphic conversion files
try:
    import PIL
except:
    print ('** PILLOW or PIL  must be installed')
    sys.exit(21)

however running below in terminal works

Code: Select all

python3 text2png.py [b]/Users/simon/Documents/iTravel/WALWATtimetable.png /Users/simon/Documents/iTravel/WALWATdepartureBoard.txt /Users/simon/Documents/PycharmProjects/iTravel/iTravel.indigoPlugin/Contents/Server\ Plugin/trainparameters.txt YES[/b]

(the bit in bold is copied from indigo logs where I print the params variable above.

I dont understand why python doesn't think that PILL is installed, when it runs standalone, and I have checked its installed with pip, pip3 and pip3.10 so not sure what version of python its trying to run this script under.

Do "which python3" in the Terminal window where you're (successfully) running the script. I wonder if you have multiple versions of Python3 installed.

Also, you can't use text formatters inside CODE tags.

Thanks. good thought.

I get this

/Library/Frameworks/Python.framework/Versions/3.10/bin/python3

When I configure a python interpreter in pycharm I get the above location as well as this one below.

Screenshot 2023-03-28 at 17.12.37.png (42.08 KiB) Viewed 1796 times

actually, I see Current is just a symlink to 3.10.

Screenshot 2023-03-28 at 17.17.56.png (120.9 KiB) Viewed 1795 times

but I do have another python3 in usr/bin which when I double click to run and try to import PIL I get an error, and that appears to be 3.8

Code: Select all

Python 3.8.2 (default, Oct  2 2020, 10:45:42)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'PIL'
>>>



[jay (support)]

I suspect you have a python install from macports or homebrew. The directories listed in your posts above are from installers that came directly from python.org (and are what the Indigo installer installs). We have an article somewhere that points out all sorts of pitfalls with having multiple python installs form multiple sources. I highly recommend that you uninstall all python versions that are not installed with Indigo, or at the very least install those versions from installers downloaded from python.org. Then they will all be in the correct directory structure and it's easy to see what's going on.

I have deliberately avoided homebrew and macports for that very reason and as I never really understood how to manage all the versions I haven't used them

The usr/bin version sounds like it's installed with xcode and is a higher version than the standard 3.1. As it (usr/bin 3. doesn't have pillow installed and 3.10 does I think that it must be using that version when called as a subprocess.

Is there anyway to specify the version to use? I am guessing somethign to do with PATH but that is where I get confused.



Or is there a more indigo specific way to have the image generation running in a separate process that won't interfere with the rest of the plugin?

[jay (support)]

The usr/bin version sounds like it's installed with xcode and is a higher version than the standard 3.1. ?

Ah, yes, I forgot there's yet one more thing that can install Python...

Is there anyway to specify the version to use?

It's always the most safest to be explicit in the path to the executable. There are tradeoffs though:

1. /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 - this is the most explicit way to start up the 3.10 interpreter. The downside is that when we (eventually) upgrade to Python 3.11, using this will then point to the wrong version
2. /usr/local/bin/python3.10 - this is a symlink to the above option, and has the same downside. In addition, though, since it's a symlink in a more public directory, it could get updated by something else entirely.
3. /Library/Frameworks/Python.framework/Versions/Current/bin/python3 - this option will overcome option 1's downside because the python 3.11 installer will change the Current component (which is a symlink) to point to the 3.11 install. The downside to this is that if you (for some reason) install an older version, the installer might update that link to point to the older version.
4. /usr/local/bin/python3 - this option is created/maintained by the python installer as well, so it has the same benefits/issues of the last option, but also since it's a symlink it could get munged by some other installer (homebrew/macports).

So, there is no completely foolproof solution. I tend to use option 3 above on my production Mac, because I never do a Python install outside of Indigo. However, I tend to use #1 above on my dev Mac because I often need multiple versions of Python installed for different scenarios and I want to be as explicit as possible.

Or is there a more indigo specific way to have the image generation running in a separate process that won't interfere with the rest of the plugin?

TBH, I don't really understand the issue. You say:

I can run the code in a normal method, but then the whole of indigo hangs when it tries to run whilst I have an edit device window open, so I am back to trying to make it run as a subprocess.

but I don't understand 1) why you would be generating an image while a dialog is open and 2) why the image generation is taking so long. So without more understanding, I can't really say if there's a better way. I would recommend taking a step back and think about the higher-level architecture of the plugin. There are almost always multiple ways to skin a cat, and it's entirely possible that a different approach would yield better results.

It's always the most safest to be explicit in the path to the executable. There are tradeoffs though:

1. /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 - this is the most explicit way to start up the 3.10 interpreter. The downside is that when we (eventually) upgrade to Python 3.11, using this will then point to the wrong version
2. /usr/local/bin/python3.10 - this is a symlink to the above option, and has the same downside. In addition, though, since it's a symlink in a more public directory, it could get updated by something else entirely.
3. /Library/Frameworks/Python.framework/Versions/Current/bin/python3 - this option will overcome option 1's downside because the python 3.11 installer will change the Current component (which is a symlink) to point to the 3.11 install. The downside to this is that if you (for some reason) install an older version, the installer might update that link to point to the older version.
4. /usr/local/bin/python3 - this option is created/maintained by the python installer as well, so it has the same benefits/issues of the last option, but also since it's a symlink it could get munged by some other installer (homebrew/macports).

So, there is no completely foolproof solution. I tend to use option 3 above on my production Mac, because I never do a Python install outside of Indigo. However, I tend to use #1 above on my dev Mac because I often need multiple versions of Python installed for different scenarios and I want to be as explicit as possible.

Or is there a more indigo specific way to have the image generation running in a separate process that won't interfere with the rest of the plugin?

TBH, I don't really understand the issue. You say:

I can run the code in a normal method, but then the whole of indigo hangs when it tries to run whilst I have an edit device window open, so I am back to trying to make it run as a subprocess.

but I don't understand 1) why you would be generating an image while a dialog is open and 2) why the image generation is taking so long. So without more understanding, I can't really say if there's a better way. I would recommend taking a step back and think about the higher-level architecture of the plugin. There are almost always multiple ways to skin a cat, and it's entirely possible that a different approach would yield better results.

Cheers…. so, this is Chameleon’s iTravel plugin that I am getting to work in python 3, I was hoping to avoid too many changes.
To be honest, I am not sure why its hanging, other than it has done repeatedly when I ran the code within the .
Agree there is no need to generate an image while a dialogue is open, but the image generation is part of the concurrent thread, it generates a fresh image every 60 seconds after its updated the travel data from teh API and written it to a text file. The generation itself is very quick, 2-3 seconds for the whole process. If I can stop that whilst its open that would be good.
Otherwise I will try and use the path to the executable for now and yes, have to update in future versions, although I thought I had tried that.

I guess I could move the image generation to a script and trigger on the text file (or a variable ) being changed, but cleaner to be self contained in a plugin.

[jay (support)]

So, that's very weird. run_concurrent_thread is run in a thread that's separate from the main plugin thread (which does dialog communication), so I'm not sure what that's hanging it up.

Maybe the PIL features that the plugin uses aren't thread-safe - that might explain it.

Ok that's interesting. I'll check that there isn't any thing running in the wrong place

So, that's very weird. run_concurrent_thread is run in a thread that's separate from the main plugin thread (which does dialog communication), so I'm not sure what that's hanging it up.

Maybe the PIL features that the plugin uses aren't thread-safe - that might explain it.

Ah run_concurrent_thread was calling a different method for the image generation within the plugin class and obviously when closing an edit device dialogue box there is another method within the plugin class that needs to run.
Maybe that's the problem.

Using the full path to python3.10 is working, the current path, and I will release with that.

The runConcurrentThread calls a method outside of the Plugin class (but still in plugin.py) which calls the API, and then that calls another method outside the Plugin class to write the image.
Would having those methods in the class keep the regular updating in its own thread,? or do I need the code in the actual runConcurrentThread method, or does none of that matter?

Thanks
Simon

ps he credits you for the subprocess shell solution in his notes, so there was obviously a reason it was implemented this way back in 2016 ish

[jay (support)]

Those functions/classes, since they are being used from RCT, will be executing in the RCT thread, so that should be fine.

This feels to me (it was 7 years ago and a major version of python back) like a workaround for a bug, probably in PIL or somesuch. Your two options (as I see it) is to limp along with what's there or, as I recommended above, take a step back and really figure out how to make this thing given the current reality (Python 3, probably significant updates to PIL, etc).