Python version conflicts

[jay (support)]

Over the years, we've seen multiple reports of issues concerning Python modules, Python versions etc. We'd like to clear this up so that when people run across issues you can understand what's going on.

First and foremost - Indigo uses the built-in Python install(s), located here:

Code: Select all

/System/Library/Frameworks/Python.framework/

Inside that directory are all the versions that macOS ships with (2.3, 2.5, 2.6, 2.7, etc.). Which ones are present is based on the OS release you're using. Since Indigo 7, Indigo has been using Python 2.7 (first added to Mac OS 10.7). Previous versions of Indigo used Python 2.5/2.6 (turns out Apple snuck in a 2.6 upgrade behind people's backs a while back, but fortunately it didn't break anything). The path to the various python executables from a shell will be:

Code: Select all

FatBook:~ jay$ which python
/usr/bin/python
FatBook:~ jay$ which easy_install
/usr/bin/easy_install

When you install a python module, you likely need to use pip (the python package manager) to install it. pip is not installed on the preinstalled Python versions so you need to install it yourself from a terminal window:

Code: Select all

 FatBook:~ jay$ sudo easy_install pip

This will install the pip command-line tool that will allow you to install all the cool modules from the Python Package Index (TONS of cool things there). Once you've installed pip, you can confirm that it's using the right Python installation:

Code: Select all

FatBook:~ jay$ which python
/usr/bin/python
FatBook:~ jay$ which pip
/usr/local/bin/pip
FatBook:~ jay$ pip -V
pip 1.4.1 from /Library/Python/2.7/site-packages/pip-1.4.1-py2.7.egg (python 2.7)

As you can see, the path to the python it what it should be (/usr/bin/python), the pip executable that you installed is in /usr/local/bin (standard location for user-installed executables), and that version of pip is pointing to Python 2.7. When you use pip to install packages, they are going into the following directory:

Code: Select all

/Library/Python/2.7/site-packages/

This is the default way Python works on macOS and how Indigo always uses Python regardless of what other 3rd party things have been done to your Mac.

The problem comes when you (or some installer you run) install a 3rd party version of Python. The three most common sources of a 3rd party Python install are: homebrew, macports, and python.org. Each will leave the existing preinstalled Pythons alone (so Indigo will continue to work), but they will insert themselves into the process such that you'll have one heck of a time getting required 3rd party modules installed in the correct spot for the preinstalled Python to see (and therefore Indigo plugins and scripts).

I believe homebrew and macports install their Python installs under a directory they create called opt (short for optional):

Code: Select all

/opt/local/Library/Frameworks/Python.framework/

and the installer from python.org will install it here:

Code: Select all

/Library/Frameworks/Python.framework/

Each of these installers will do several things:

1. They will insert paths into the various .login, .profile, .bashrc files such that their version of Python (and it's executables) are found first when typing any of the python commands from the shell (they also likely install their own pip)
2. They set the PYTHONPATH (which is a list of file system paths that the Python interpreter looks in to find modules not part of the standard install) so that they look in their own site-packages directories

Let's start with the second one first: they will add their own site-packages directories (i.e. /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages) as well as the default one (/Library/Python/2.7/site-packages/) to PYTHONPATH. This will, on the surface, make it seem that they're working OK because they're finding things that may have been installed prior to their installation.

The problem is that when you install further modules using pip (the one they installed for you) those modules will get installed into their site-packages directory rather than the one used by the preinstalled Python (/Library/Python/2.7/site-packages/). So anything new you install won't be accessible from the preinstalled Python that Indigo uses. But when you test from a shell script, using the python command, it works. This is because that command isn't using the preinstalled Python. This duality makes it very confusing to figure out what's going on because it appears to work from a shell but not from Indigo.

It is possible that you can manage having multiple Python installs - you just have to remember which paths to executables you need to use to get modules installed in the correct place. However, this is not for the faint of heart.

We highly recommend that you do not install 3rd party Pythons. This will make sure that your Indigo experience is as painless as possible.

This is a duplicate of another post - if you have any comments post on that topic (this one is locked to avoid duplicate posts).

[rhanson]

How about having an option in the config file to point to a virtualenv path so Indigo knows what to use? That would allow us to keep the native macOS python directory clean (and unused).

Or is that what you meant by not for the faint of heart?

[FlyingDiver]

That would allow us to keep the native macOS python directory clean (and unused).

Why would you want/need to do that?

[jay (support)]

How about having an option in the config file to point to a virtualenv path so Indigo knows what to use? That would allow us to keep the native macOS python directory clean (and unused).

Or is that what you meant by not for the faint of heart?

That doesn't really solve the problem because it would be yet another place users would have to install modules (and in yet another way) thereby adding even more confusion. The issue isn't one of conflicting module versions (which virtualenv would help somewhat by isolating it) but rather installing the module in the correct place for Indigo to see.

[midd]

SO I have discovered that I have fallen victim to this. I have the Python.org version of 2.7.2 installed. There's probably no way of uninstalling it without doing damage to Indigo? Or would reinstalling Indigo be a way of correcting it?

[jay (support)]

SO I have discovered that I have fallen victim to this. I have the Python.org version of 2.7.2 installed. There's probably no way of uninstalling it without doing damage to Indigo? Or would reinstalling Indigo be a way of correcting it?

Their installer may have an uninstall option when you run it though I don't know for sure. Reinstalling Indigo won't help because Indigo isn't the problem.

[midd]

ok. I'm having a helluva time installing pytz, pip and easy_install. Here are my list of pythons to choose from:

Code: Select all

server:~me$ port select --list python
Warning: port definitions are more than two weeks old, consider updating them by running 'port selfupdate'.
Available versions for python:
	none
	python25-apple
	python26-apple
	python27
	python27-apple (active)

I had MacPorts version of python installed. I'm pretty sure I uninstalled it today. Just figured out that python27 was installed for Webdeck's Homebridge configuration. Deleted the ports used and then deleted python27. Hoping this works out!

[FlyingDiver]

easy_install is part of the Apple default Python install. You should not need to install it. Use it to install pip. Then use pip to install anything after that.

[midd]

I have been trying. Typed sudo easy_install pip and here's the response:

Code: Select all

Searching for pip
Reading http://pypi.python.org/simple/pip/
Couldn't find index page for 'pip' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for pip
error: Could not find suitable distribution for Requirement.parse('pip')

when I copy and paste that pypi.python.org address into my web browser, it's a secured site (https://pypi.python.org). Would that be the reason terminal can't find any packages or links for pip?

[GlennNZ]

Hi

As per PM - which hopefully you’ll get when next around.

I think the best way forward is to reinstall easy_install to fix the likely path issues.

Delete here: (May not need to delete)
Welcome to start with install and see whether that fixes easy_install

Code: Select all

sudo rm -f /usr/bin/easy_install*
sudo rm -f /usr/local/bin/easy_install*

& then reinstall it

Code: Select all

curl -O https://svn.apache.org/repos/asf/oodt/tools/oodtsite.publisher/trunk/distribute_setup.py
sudo python distribute_setup.py
sudo rm distribute_setup.py

Or can install pip alone and use that with:

Code: Select all

curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py

[midd]

Got it working finally! whew! Jay is right. It's a nightmare. But all of the third party Pythons are gone. Thanks @GlennNZ for sticking it out with me!

[TwitchCaptain]

Hello guys,

I'm a little confused. I just installed a fresh 10.12 OS on my server, and attempted to do

Code: Select all

sudo easy_install pexpect 

- pexpect is a library I use in one of my plugins. But I've come to realize easy_install no longer works on macOS because of an outdated openSSL library. This same error happens with easy_install pip as well.

Code: Select all

$ sudo easy_install pexpect
Searching for pexpect
Reading https://pypi.python.org/simple/pexpect/
Download error on https://pypi.python.org/simple/pexpect/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) -- Some packages may not be found!
Couldn't find index page for 'pexpect' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) -- Some packages may not be found!
No local packages or download links found for pexpect
error: Could not find suitable distribution for Requirement.parse('pexpect')

What is the supported method for installing python libraries our Indigo plugins can use?

EDIT: I'd just like to point out that the Internet provides lots of "unsupported" ways to fix this. I'd like to hear how Perceptive Automation is approaching this problem. Upgrading to 10.13 is simply not an option (for a number of reasons, but especially because applescript handler support was removed from messages.app). This is one of the "solutions" I found, but I'm not messing with my system python until someone from support chimes in to tell me to do so. https://stackoverflow.com/questions/497 ... tisfies-th - I've had way too many headaches with python, so I avoid touching it as much as possible. At this point, it's a fresh system, all untouched.

Thanks!

[jay (support)]

We don't have an answer to the TLS problem because there's really nothing we can do. We're entirely dependent on the Python installation provided by macOS.

Having said that, the pip workaround in that post might very well work.

And - we generally recommend that if you're going to distribute a plugin that you include the module in the plugin itself. This should work fine for modules that are all Python. For modules that contain compiled source, it's more of a challenge.

[FlyingDiver]

The solution in that thread works, with one caveat. You can't so it on one command line with sudo. You need to do:

Code: Select all

sudo -s
curl https://bootstrap.pypa.io/get-pip.py | python
pip install pexpect

Apple should fix this by putting out a patch for 10.12 with an updated pip in it. But I don't think they will. So your only real option is to fix it yourself. The commands above will do that just fine. The alternative, which is much more work, is to use pip on a 10.13 machine then move the files over to your 10.12 machine manually.

[TwitchCaptain]

I'll hack in a pip upgrade for now, and then figure out how to bundle libs next. I think someone said they did it with one of my other plugins in a fork, so I'll start there. Thanks for the info!