Is the Python requests module broken on Catalina?

If you have Catalina, try running this in Terminal:

Code: Select all

python -c 'import requests; req=requests.get("https://www.apple.com/"); print req.status_code'


Seems pretty straightforward, right? No, not really.

Code: Select all

jkeenan@Kraken ~ % python -c 'import requests; req=requests.get("https://www.apple.com/"); print req.status_code'
From cffi callback <function _verify_callback at 0x104d30c80>:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/OpenSSL/SSL.py", line 309, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)


On Mojave, no problem:

Code: Select all

home:~ miniserver$ python -c 'import requests; req=requests.get("https://www.apple.com/"); print req.status_code'
200


On Catalina, with Python3. Note the syntax change for the print statement.

Code: Select all

jkeenan@Kraken ~ % python3 -c 'import requests; req=requests.get("https://www.apple.com/"); print(req.status_code)'
200


This has bitten me on three different plugins this weekend. The workaround, for now, is to disable certificate checking:

Code: Select all

jkeenan@Kraken ~ % python -c 'import requests; req=requests.get("https://www.apple.com/", verify=False); print req.status_code'
/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
200


Warning messages, but at least it runs.

I'm documenting my research into this issue in this thread, so if someone else runs into this they'll know what's going in.

Also, I confirmed this exact same behavior on my MBP, which was also upgraded to Catalina. Both systems were upgrades from Mojave (and others before that). So I decided to see what happens on a clean install of Catalina. For that, I used a VM in Parallels.

Python 2.7.16 is pre-installed in Catalina (Mojave was 2.7.10). Catalina also has Python 3.7.3 pre-installed.

Neither pip nor the requests module are pre-installed for Python 2. Pip is installed for Python 3. I ran the following commands:

Code: Select all

sudo easy_install pip
sudo pip install requests


And what do you know - the test script worked!

The pip install requests command installed the following packages:

certifi-2019.9.11
chardet-3.0.4
idna-2.8
requests-2.22.0
urllib3-1.25.6

My upgraded Catalina systems had:

certifi-2017.7.27.1
chardet-3.0.4
idna-2.6
requests-2.11.1
urllib3-1.22

What wasn't listed as installed by pip, but I believe is crucial based on researching the error message, is the cryptography package. Which is version 2.6.1 in Catalina, but version 1.72. on my upgraded systems.

So here's what I did to fix it:

Code: Select all

sudo pip uninstall cryptography certifi requests urllib3
pip list   # check to make sure the packages are actually not in the list anymore.  I had to repeat a couple.
sudo pip install requests


Then I ran the test script again:

Code: Select all

jkeenan@Kraken ~ % python -c 'import requests; req=requests.get("https://www.apple.com/"); print req.status_code'
WARNING: Executing a script that is loading libcrypto in an unsafe way. This will fail in a future version of macOS. Set the LIBRESSL_REDIRECT_STUB_ABORT=1 in the environment to force this into an error.
200


I only get that warning on one of my systems. The other is warning free. Damifino. But it works.

Joe - thanks for the research. Really appreciated. Based on your research, do you think this means that any plugin that uses requests will fail for users that have upgraded to Catalina (but have not applied the steps above)? If the answer is yes, I wonder if there's a way to include a "repaired" requests library within a plugin package that will work for Catalina users, but also work for users of prior OSes. Or alternatively, I wonder if it's possible for Indigo to include a "repaired" version that could be called by all plugins.

Pseudocode:

Code: Select all

If mac_os < 10.15:
    import requests (old)
else:
    import requests (new)


This sounds like it has the potential to be quite the cluster.

Bonus pain: at some point we'll also have to contend with making plugins simultaneously compatible with Python 2.x and 3.x as well.

I thought the Requests module is included in the Indigo distribution?

I thought the Requests module is included in the Indigo distribution?

It is, now. Starting in 7.0 I think.

Did you try my test in the first post? Did it fail for you?

Joe - thanks for the research. Really appreciated. Based on your research, do you think this means that any plugin that uses requests will fail for users that have upgraded to Catalina (but have not applied the steps above)? If the answer is yes, I wonder if there's a way to include a "repaired" requests library within a plugin package that will work for Catalina users, but also work for users of prior OSes. Or alternatively, I wonder if it's possible for Indigo to include a "repaired" version that could be called by all plugins.

Pseudocode:

Code: Select all

If mac_os < 10.15:
    import requests (old)
else:
    import requests (new)


This sounds like it has the potential to be quite the cluster.

First, I don't know how widespread this is. Maybe there's only a problem when all these are installed manually. Do you have the problem on any of your systems?

Most users will not have ever installed requests on their own, since it's bundled with Indigo. In which case bundling up-to-date versions of certifi and cryptography might eliminate the problem.

I was really hoping for more people to run the test so we have some data on how common this is.

Bonus pain: at some point we'll also have to contend with making plugins simultaneously compatible with Python 2.x and 3.x as well.

I figure when Indigo 8 (or whatever) includes a Python run time, it'll be Python 3.x and plugins for 8+ will need to work with it. Maybe the store can be enhanced to download the new versions for 8 and older version for 7.

Unfortunately, I haven't tested it since I don't have a production copy of Catalina installed anywhere.

If you think it would be helpful, I could try it against a Catalina beta that I have installed in a test machine.

[jay (support)]

If you have Catalina, try running this in Terminal:

Code: Select all

python -c 'import requests; req=requests.get("https://www.apple.com/"); print req.status_code'


I just ran this in a couple of different scenarios. First, from within the Python Scripting Shell from Indigo 7.4, which is what most people will experience:

Code: Select all

>>> import requests; req=requests.get("https://www.apple.com/"); print req.status_code
200


So the bundled requests (v2.10.0) with Indigo still works correctly, assuming there is a clean Python 2 install:

Code: Select all

>>> import requests
>>> requests.__version__
'2.10.0'
>>>

Second, exactly as you have above:

Code: Select all

admin@Mac-mini ~ % python -c 'import requests; req=requests.get("https://www.apple.com/"); print req.status_code'
\Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named requests
admin@Mac-mini ~ %


This is as expected since Catalina's bundled Python 2 doesn't include requests. For grins, I ran this:

Code: Select all

python3 -c 'import requests; req=requests.get("https://www.apple.com/"); print req.status_code'

And received the dialog that python3 requires command line developer tools, and prompted if I wanted to install. Given that I want to keep this Mac a "clean" install that most matches what users are likely to have, I declined. But it wouldn't have worked either unless Apple installs requests for Python 3 (which seems doubtful).

So maybe this is only going to bite developers who have used pip to install modules outside of Indigo.