Add dynamically linked musl distributions #541

zanieb · 2025-02-26T04:34:27Z

As a consequence of #540 I was playing with these concepts and decided to explore it.

This includes #546 (could be merged separately or together), which separates "static" builds from the "musl" triple specifically in favor of a dedicated build option.

The main implementation downside here is that $ORIGIN doesn't work with DT_NEEDED so we need to use RUNPATH instead, which can cause the wrong library to be loaded if LD_LIBRARY_PATH is set. Given the current builds aren't usable at all, I think this is a fine trade-off. We can explore alternatives in the future, like statically linking just libpython.

Another caveat here: for consistency with the glibc builds, we're changing the "default" musl build to be dynamically linked. This is a breaking change in the release artifacts. The statically linked musl build will include a +static suffix. We could do something for backwards compatibility here, but I think this probably makes sense in the long term. My primary concern is that consumers that combine releases (such as uv) would need to encode this change (e.g., toggle the expectation based on the python-build-standalone version tag).

It's challenging to test changes to the release artifact handling. Regardless of approach, this will need a follow-up to adjust that accordingly.

cpython-unix/build.py

zanieb · 2025-02-26T04:47:19Z

I got the x86_64-unknown-linux-musl-noopt build passing on my machine and gave it a quick test in an alpine container and it looks like I didn't package it right.

/test # ldd ./python/install/bin/python
	/lib/ld-musl-x86_64.so.1 (0x7ebb73af0000)
Error loading shared library $ORIGIN/../lib/libpython3.13.so.1.0: No such file or directory (needed by ./python/install/bin/python)
	libc.so => /lib/ld-musl-x86_64.so.1 (0x7ebb73af0000)
Error relocating ./python/install/bin/python: Py_BytesMain: symbol not found

I presumed we just didn't package the .so, but it is there

/ # ls /test/python/install/lib/
Tix8.4.3              libpython3.13.so      pkgconfig             tcl8.6
itcl4.2.4             libpython3.13.so.1.0  python3.13            thread2.8.9
libpython3.13.a       libpython3.so         tcl8                  tk8.6

So I futzed with patchelf

/ # patchelf --set-rpath '/test/python/install/lib' /test/python/install/bin/python
/ # ldd /test/python/install/bin/python
	/lib/ld-musl-x86_64.so.1 (0x79dc36f38000)
Error loading shared library $ORIGIN/../lib/libpython3.13.so.1.0: No such file or directory (needed by /test/python/install/bin/python)
	libc.so => /lib/ld-musl-x86_64.so.1 (0x79dc36f38000)
Error relocating /test/python/install/bin/python: Py_BytesMain: symbol not found

Huh

/ # readelf -d /test/python/install/bin/python

Dynamic section at offset 0x388 contains 21 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [/test/python/install/lib]
 0x0000000000000001 (NEEDED)             Shared library: [$ORIGIN/../lib/libpython3.13.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000c (INIT)               0x401000
 0x000000000000000d (FINI)               0x401148
 0x0000000000000019 (INIT_ARRAY)         0x403e48
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x403e50
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x3ff468
 0x000000006ffffef5 (GNU_HASH)           0x3ff428
 0x0000000000000005 (STRTAB)             0x3fd2e0
 0x0000000000000006 (SYMTAB)             0x3ff338
 0x000000000000000a (STRSZ)              167 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x403fe8
 0x0000000000000002 (PLTRELSZ)           48 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x400470
 0x0000000000000000 (NULL)               0x0

I guess I'll try harder?

/ # patchelf --replace-needed '$ORIGIN/../lib/libpython3.13.so.1.0' '/test/python/install/lib/libpython3.13.so.1.0' /test/pyt
hon/install/bin/python
/ # /test/python/install/bin/python --version
Python 3.13.2

Seems like $ORIGIN probably isn't supported? Need to look into that.

zanieb · 2025-02-26T17:08:29Z

ORIGIN is supported for RPATH, but seemingly not for NEEDED (ref https://git.musl-libc.org/cgit/musl/tree/ldso/dynlink.c#n897) — switching to that for now.

This was very opaque in the error message in CI

This is what it is available on ubuntu-22.04

zanieb · 2025-02-26T19:46:41Z

Hm okay something is wrong with Python 3.11 — peachy otherwise!

edit: It's loading the libpython from the system, which is 3.11.

zanieb · 2025-02-26T20:45:42Z

This is loosely ready for review. I have some minor clean-ups and documentation to do.

indygreg

I strongly prefer retaining the ability to produce a statically linked musl distribution. There are some users of PyOxidizer who really like single file binaries. And you only get that if you statically link musl.

I would prefer standing up a new target variant with dynamic linked musl which is ABI compatible with the musl Python platform/target.

zanieb · 2025-02-27T15:19:23Z

I can look into that. Will require #542.

# Conflicts: # cpython-unix/build-cpython.sh # cpython-unix/build.py # pythonbuild/cpython.py

zanieb · 2025-03-06T19:50:46Z

cpython-unix/build.py

-            # TODO should the musl target be normalized?
-            .replace("-unknown-linux-musl", "-unknown-linux-gnu")


See xproto and xextproto build changes

The answer to this todo seems to be: no it should not generally be normalized like this because it causes problems. But then there are a couple specific builds where it does need to be normalized because there isn't support for -musl targets.

See also, the tcl build change.

… definition error

zanieb · 2025-03-07T14:25:11Z

pythonbuild/downloads.py

+        "url": "https://musl.libc.org/releases/musl-1.2.2.tar.gz",
+        "size": 1055220,
+        "sha256": "9b969322012d796dc23dda27a35866034fa67d8fb67e0e2c45c913c3d43219dd",
+        "version": "1.2.2",


We use an older version for dynamic builds, for compatibility with more systems. For example, the GitHub runners we're using did not have 1.2.5.

We retain the latest version for static builds, since compatibility isn't a concern.

geofft

LGTM. At first glance I'm okay changing the default musl build to be dynamic and to make the static variant an explicit option, and I would assume that works fine with the release process since it at most adds more artifacts, some of which might get missed, but the existing set of artifacts will still be present (with a semantics change for the musl ones). But we should probably double-check whether uv and Rye are chill with the change before shipping. Are there any other high-profile users we should check, especially ones we know are using the musl builds? (PyOxidizer?)

geofft · 2025-03-10T18:37:47Z

cpython-unix/build-libX11.sh

+    mipsel-unknown-linux-musl)
+      EXTRA_FLAGS="${EXTRA_FLAGS} --enable-malloc0returnsnull"
+      ;;


I'm assuming we were previously implicitly doing this for musl?

cpython-unix/build-xextproto.sh

geofft · 2025-03-10T18:48:03Z

src/validation.rs

+        allowed_libraries.push(format!("libpython{}t.so.1.0", python_major_minor));
+        allowed_libraries.push(format!("libpython{}td.so.1.0", python_major_minor));
+    }
+


The changes from line 921 to here seem a little inconsistent, are all the changes here intentional?

Oh thanks the changes around 924 are confusing? I'll revert that.

I can also probably make this bit conditional on the static build option.

zanieb · 2025-03-10T20:16:06Z

Are there any other high-profile users we should check, especially ones we know are using the musl builds? (PyOxidizer?)

I want to ack this, but I don't have a good answer. @ofek comes to mind as an automated consumer outside Astral.

ofek · 2025-03-10T20:28:01Z

Changing the default to something that actually works is desirable in my opinion. It doesn't matter if there is new naming because in my projects that consume (PyApp & Hatch) there is no ability to use newer non-custom builds without a new release because PyApp has a script that generates the hardcoded source of truth, which Hatch then uses in a script to hardcode its source of truth. So I would just have to change the former script.

zanieb · 2025-03-11T14:26:55Z

I considered a transition plan like

Continue to publish static builds as we are today
Create a duplicate archive with +static
Publish dynamic builds with +shared
Switch the default to the +shared variant, retaining a +shared duplicate archive
Drop the +shared archive

However, users still need to encode a specific version of python-build-standalone in which the default changed. After thinking about it more, I'm still leaning towards just changing the defaut.

geofft · 2025-03-11T16:05:05Z

@ofek by "actually works" I assume you mean "able to dynamically load stuff"?

Then I think we should ship this and call it a bugfix and make sure we ship it at a time that people will be around on Discord etc. to help downstream folks that are broken by the change if any are actually broken by it.

We could also figure out how to make a static musl include a working libdl, which is probably easier than it would be for glibc because musl is generally less complex, but would still be a little bit of a research project.

zanieb · 2025-03-11T16:50:46Z

The diff is not too complicated, so I'll just merge this and close #546

Follows #541 Tested locally with `just release-dry-run` and the artifacts on the latest commit to `main`. Then used for https://github.com/astral-sh/python-build-standalone/releases/tag/20250311

Dynamically link libc in musl distributions

9663760

zanieb added the libc:musl label Feb 26, 2025

zanieb commented Feb 26, 2025

View reviewed changes

cpython-unix/build.py Outdated Show resolved Hide resolved

zanieb added the platform:linux Specific to the Linux platform label Feb 26, 2025

zanieb added 3 commits February 26, 2025 10:36

Fix dynlib loading

18ff4fc

Update musl dist metadata

f156f44

Update validation to support dynamic musl

c78bb9b

zanieb added 5 commits February 26, 2025 11:09

Update comment

d590f87

Fix verify test for dynamic musl

51a5e29

Add path context to test run failure

7b2b0f2

This was very opaque in the error message in CI

Format

2cef305

Install musl-dev for validation

a173de8

zanieb mentioned this pull request Feb 26, 2025

Refactor static and dynamic handling to avoid hard-coding musl #542

Closed

zanieb added 2 commits February 26, 2025 12:55

Fix musl build mode loading

94cadca

Downgrade to musl 1.2.2

7b8ff31

This is what it is available on ubuntu-22.04

zanieb added 2 commits February 26, 2025 14:22

Debug LD_LIBRARY_PATH

280f345

Add comment

902fcd0

zanieb marked this pull request as ready for review February 26, 2025 20:45

zanieb requested a review from geofft February 26, 2025 20:55

indygreg reviewed Feb 27, 2025

View reviewed changes

This was referenced Feb 27, 2025

Non Portable binary, doesn't work with sudo or without LD_LIBRARY_PATH actions/setup-python#871

Open

Refactor static / dynamic linking into build options #546

Closed

zanieb added 3 commits March 5, 2025 13:37

Refactor static / dynamic linking into build options

b60e292

Merge branch 'zb/ref-static' into zb/musl-dynamic

28d5d6e

# Conflicts: # cpython-unix/build-cpython.sh # cpython-unix/build.py # pythonbuild/cpython.py

Add dynamic CI targets

01172ad

zanieb added 4 commits March 6, 2025 13:12

Fix syntax error

6f88ed4

Set -x in the musl build

c0b9ce9

Only enable fPIC during shared musl builds

44c1959

Trim some extraneous whitespace

4385eb8

zanieb commented Mar 6, 2025

View reviewed changes

zanieb added 6 commits March 6, 2025 14:35

Change the triple during static musl tcl builds to get past duplicate…

1990014

… definition error

Format

7153487

Fix typo

4c685db

Fix missing variable

4670133

Fix configuration of tcl build

c4e081d

Update validation

c410304

zanieb commented Mar 7, 2025

View reviewed changes

zanieb marked this pull request as ready for review March 10, 2025 16:38

geofft approved these changes Mar 10, 2025

View reviewed changes

zanieb added 2 commits March 10, 2025 15:01

Remove extra "

a745841

Clean up ELF validation logic

6a79ece

This comment was marked as duplicate.

Sign in to view

geofft approved these changes Mar 11, 2025

View reviewed changes

zanieb merged commit 08c1fac into main Mar 11, 2025
134 checks passed

zanieb deleted the zb/musl-dynamic branch March 11, 2025 16:50

zanieb mentioned this pull request Mar 11, 2025

Update release artifacts for musl to include +static variant #556

Merged

This was referenced Mar 12, 2025

MUSL distributions lacking proper interpreter information #260

Closed

Allow dynamic loading in musl builds while CPython itself is static #86

Closed

Musl mise binary will install musl-linked static python jdx/mise#3295

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dynamically linked musl distributions #541

Add dynamically linked musl distributions #541

zanieb commented Feb 26, 2025 •

edited

Loading

zanieb commented Feb 26, 2025

zanieb commented Feb 26, 2025

zanieb commented Feb 26, 2025 •

edited

Loading

zanieb commented Feb 26, 2025

indygreg left a comment

zanieb commented Feb 27, 2025

zanieb Mar 6, 2025

zanieb Mar 7, 2025

zanieb Mar 7, 2025

geofft left a comment

geofft Mar 10, 2025

geofft Mar 10, 2025

zanieb Mar 10, 2025 •

edited

Loading

zanieb Mar 10, 2025

This comment was marked as duplicate.

zanieb commented Mar 10, 2025

ofek commented Mar 10, 2025

zanieb commented Mar 11, 2025

geofft commented Mar 11, 2025

zanieb commented Mar 11, 2025

		# TODO should the musl target be normalized?
		.replace("-unknown-linux-musl", "-unknown-linux-gnu")

Add dynamically linked musl distributions #541

Add dynamically linked musl distributions #541

Conversation

zanieb commented Feb 26, 2025 • edited Loading

zanieb commented Feb 26, 2025

zanieb commented Feb 26, 2025

zanieb commented Feb 26, 2025 • edited Loading

zanieb commented Feb 26, 2025

indygreg left a comment

Choose a reason for hiding this comment

zanieb commented Feb 27, 2025

zanieb Mar 6, 2025

Choose a reason for hiding this comment

zanieb Mar 7, 2025

Choose a reason for hiding this comment

zanieb Mar 7, 2025

Choose a reason for hiding this comment

geofft left a comment

Choose a reason for hiding this comment

geofft Mar 10, 2025

Choose a reason for hiding this comment

geofft Mar 10, 2025

Choose a reason for hiding this comment

zanieb Mar 10, 2025 • edited Loading

Choose a reason for hiding this comment

zanieb Mar 10, 2025

Choose a reason for hiding this comment

This comment was marked as duplicate.

zanieb commented Mar 10, 2025

ofek commented Mar 10, 2025

zanieb commented Mar 11, 2025

geofft commented Mar 11, 2025

zanieb commented Mar 11, 2025

zanieb commented Feb 26, 2025 •

edited

Loading

zanieb commented Feb 26, 2025 •

edited

Loading

zanieb Mar 10, 2025 •

edited

Loading