Homebrew + Apple Silicon + OpenSSL 1.1/3 + Python compile issues

[nickwilliams-zaxiom]

Output of brew config

Apple Silicon Machine:
HOMEBREW_VERSION: 4.3.10
ORIGIN: https://github.com/Homebrew/brew
HEAD: c3ae0327c01aab461531bfc0fcdfb4dd3a779205
Last commit: 23 hours ago
Core tap JSON: 18 Jul 22:04 UTC
Core cask tap JSON: 18 Jul 22:04 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 3.3.3 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.3.3/bin/ruby
CPU: dodeca-core 64-bit arm_blizzard_avalanche
Clang: 15.0.0 build 1500
Git: 2.45.2 => /opt/homebrew/bin/git
Curl: 8.6.0 => /usr/bin/curl
macOS: 14.5-arm64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.4
Rosetta 2: false

Intel Machine:
HOMEBREW_VERSION: 4.3.9
ORIGIN: https://github.com/Homebrew/brew
HEAD: 1c98a33a376e5b35ebc801d0f7d6ba7b9eb118ac
Last commit: 12 days ago
Core tap JSON: 17 Jul 21:52 UTC
Core cask tap JSON: 17 Jul 21:52 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 16
Homebrew Ruby: 3.3.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.3/bin/ruby
CPU: 16-core 64-bit kabylake
Clang: 15.0.0 build 1500
Git: 2.45.2 => /usr/local/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.4-x86_64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.4

Output of brew doctor

Apple Silicon Machine:
Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  [email protected]

Warning: Broken symlinks were found. Remove them with `brew cleanup`:
  /opt/homebrew/opt/python@3

Intel Machine:
Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  [email protected]
  [email protected]

Warning: Homebrew's "sbin" was not found in your PATH but you have installed
formulae that put executables in /usr/local/sbin.
Consider setting your PATH for example like so:
  echo 'export PATH="/usr/local/sbin:$PATH"' >> /Users/williamsn/.bash_profile

Description of issue

Note 1: I can reliably reproduce this issue on my Apple Silicon MacBook Pro running Sonoma 14.5 and Xcode 15.4. I can reliably not reproduce this issue on my Intel MacBook Pro running Sonomal 14.5 and Xcode 15.4. It's possible that the problem is actually directory paths and not Apple vs. Intel, because for some reason Homebrew stuff is installed at /opt/homebrew on my Apple Silicon machine and at /usr/local on my Intel machine (which had Homebrew installed 3 years before the Apple Silicon machine).

Note 2: This might not be the correct place for this. I have no idea if this is a Homebrew bug, a pyenv formulae bug, or an outright pyenv bug. Apologies if I have put this in the wrong place.

Background: Because of the work I do, there are conflicts with using OpenSSL 3 + Python on my machine. When I'm in my development environment, if Python is linked against OpenSSL 3, I can't import the Python ssl module because of conflicts with a dev-environment compilation of OpenSSL 3 that is earlier on the dev-environment path. As such, I have a need to compile Python against OpenSSL 1.1 so that I can use Python in my development environment. And that's how I discovered this problem...

I have this exported so that, whenever pyenv builds Python, it explicitly links against formulae [email protected] instead of defaulting to openssl@3:

export [email protected]

This works fine on my Intel machine. It does not work (on its own) on my Apple Silicon machine. The tl;dr is that the command results in a Python installation where _ssl.OPENSSL_VERSION reports 1.1.1w on my Intel machine, but on my Apple Silicon machine causes compile "errors" about _ssl not being importable and so being removed from the Python installation, and yet when I use that Python installation I can import _ssl, but _ssl.OPENSSL_VERSION reports 3.3.1:

pyenv install --verbose 3.8.19

However, this command on my Apple Silicon machine results in _ssl.OPENSSL_VERSION reporting 1.1.1.w:

LDFLAGS="-L/opt/homebrew/opt/[email protected]/lib" CPPFLAGS="-I/opt/homebrew/opt/[email protected]/include" pyenv install --verbose 3.8.19

To get more detailed, here are some relevant excerpts from the logs on the Apple Silicon machine without setting the environment variables:

python-build: use [email protected] from homebrew
...
checking for openssl/ssl.h in /opt/homebrew/opt/[email protected]... yes
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
...
clang ... -DOPENSSL_NO_SSL3 ...
...
*** WARNING: renaming "_ssl" since importing it failed: dlopen(/private/var/folders/30/nyd4lxbs5y15ngpjpl441m4w0000gn/T/python-build.20240719102758.54761/Python-3.10.14/build/lib.macosx-14.5-arm64-3.10/_ssl.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_SSL_get_peer_certificate'
...
Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl

Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer
Custom linker flags may require --with-openssl-rpath=auto

Based on that output, I would expect that I would not be able to import _ssl. However, I can, just with unexpected results:

>>> import _ssl
>>> _ssl.OPENSSL_VERSION
'OpenSSL  3.3.1 4 Jun 2024'

There is no log output that states that _ssl got recompiled against the openssl@3 formulae, but somehow it did.

With the two environment variables, the "renaming" and "but were removed" sections go away, and:

>>> import _ssl
>>> _ssl.OPENSSL_VERSION
'OpenSSL 1.1.1w  11 Sep 2023'

And, again, I don't need those environment variables on my Intel machine. It "just works" and reports OpenSSL 1.1.1.w.

In all the trial and error, I have lost my original full build logs; however, if the problem cannot be easily seen from the information I provided, I can uninstall these Python versions from pyenv and reinstall the various ways in order to capture full build logs.