Allow stable ABI extensions to include multiarch tuples in the filename

[stefanor]

Feature or enhancement

Proposal:

Currently C extensions can include the multarch tuple of the architecture the support in their filename, e.g. zlib.cpython-314-x86_64-linux-gnu.so. This extensions for multiple different architectures to be installed on the same filesystem, next to each other, without clashing. This is used by Debian to allow packages for multiple architectures to be co-installed simultaneously.

The same should apply to stable ABI C extensions, but this was missed when the stable ABI was implemented.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-122931 Allow stable API extensions to include a multiarch tuple in the filename #122917

[rgommers]

This will not be so easy to introduce. Currently build systems/backends get the cpython-314-x86_64-linux-gnu.so extension name from a value like sysconfig.get_config_var('EXT_SUFFIX') while they hardcode abi3.so because it's the only option.

I assume you want all build tools to do the new thing and produce abi3-x86_64-linux-gnu.so always instead of abi3.so? There are a lot of build tools to update then.

[stefanor]

This will not be so easy to introduce.

The plan was: Start supporting this as an alternative, until we reach a point where we don't have to support older versions that don't support it any more. Then it can become the default.

We have supported this in Debian since 3.13, and were hoping to get support upstream into 3.14.0, but moved a little too late for that :(

I assume you want all build tools to do the new thing and produce abi3-x86_64-linux-gnu.so always instead of abi3.so? There are a lot of build tools to update then.

If build tools are asking the interpreter what to prefer, then they should do the right thing. Which is to not build anything with this, for now.

We handle renaming extensions to include this path in our package tooling. Nothing else really needs it, at this point.

[rgommers]

We handle renaming extensions to include this path in our package tooling. Nothing else really needs it, at this point.

Ah great, thanks for the context. No concerns from me then.

[eli-schwartz]

This will not be so easy to introduce. Currently build systems/backends get the cpython-314-x86_64-linux-gnu.so extension name from a value like sysconfig.get_config_var('EXT_SUFFIX') while they hardcode abi3.so because it's the only option.

This is of course not true. Meson uses importlib.machinery.EXTENSION_SUFFIXES[1] on the theory that element 0 is the the EXT_SUFFIX (because it is most specific and thus most preferred) while element 1 is the second-most preferred which ought to be the stable ABI suffix.

Inserting another abi3 suffix so that it looks like

['.cpython-314-x86_64-linux-gnu.so', '.abi3-x86_64-linux-gnu.so', '.abi3.so', '.so']

is:

• intrinsically necessary (to include in the list) to get the importlib machinery to work correctly
• intrinsically necessary (to use that specific order) because an architecture specific abi3 extension needs to take priority over an abi3 extension of unknown compatibility with the current interpreter
• automatically going to work for meson

The plan was: Start supporting this as an alternative, until we reach a point where we don't have to support older versions that don't support it any more. Then it can become the default.

This is an unnecessary complication since both names have to be supported for compatibility, so the default can immediately become the "better" one. Tools that don't know how to use the better one don't have to, because for compatibility they both work! Tools that do.know how to use the better one... should do so?

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter. Rationale is that the minimum limited API setting being lower than the current headers version, "doesn't actually guarantee you're avoiding stable ABIs from the most recent version", and shall not be supported, therefore it's useless to say you'd like to be able to do it.

[stefanor]

Meson uses importlib.machinery.EXTENSION_SUFFIXES[1]

That does sound a little fragile. It select take the first one with .abi in it, and be a little more resilient to change. setuptools does that:
https://github.com/pypa/setuptools/blob/d198e86f57231e83de87975c5c82bc40c196da79/setuptools/command/build_ext.py#L79-L86

intrinsically necessary (to use that specific order) because an architecture specific abi3 extension needs to take priority over an abi3 extension of unknown compatibility with the current interpreter

I don't think we want that behaviour until we can assume that any interpreter that is able to get this extension in a wheel can import this filename.

This is an unnecessary complication since both names have to be supported for compatibility, so the default can immediately become the "better" one.

Same issue. If you're building any binaries that other people are going to use with other interpreters, you need to ensure that they can import them.

[eli-schwartz]

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter.

[eli-schwartz]

Meson uses importlib.machinery.EXTENSION_SUFFIXES[1]

That does sound a little fragile.

It is! It doesn't work in cross compilation! Changing this would be lovely, but playing games with modifying the heuristics about how to parse details of a runtime import of the importlib machinery are very uninteresting.

In fact it's already been changed -- if you use a PEP 739 build_details.json, the abi.stable_abi_suffix is mandatory for CPython (and any other python implementations that have a stable ABI) and is/can be used in preference to guessing anything.

One would presume therefore that that build_details.json would list the intended ABI suffix (the one that is most specific to the current interpreter, i.e. the one that lists the CPU arch).

[stefanor]

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter.

IIRC that's a new thing, since we made this proposal. And there's nothing to declare the compatible versions in the data...

Also, right now if you compile on a Debian Python 3.13, the multiarch extension is there...

[eli-schwartz]

And there's nothing to declare the compatible versions in the data...

@zooba specifically wrote PEP 809 to address this very point -- the stable ABI would permit neither additions nor removals except when revbumping the ABI tag (abi4, abi2026, depending).

Also, right now if you compile on a Debian Python 3.13, the multiarch extension is there...

If the idea is to add this change to the official unpatched interpreter, this problem doesn't exist.

[merwok]

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter.

IIRC that's a new thing, since we made this proposal.

No, that’s how the stable ABI works: https://docs.python.org/3/c-api/stable.html#limited-c-api

You can compile with limited API version 3.12 using Python 3.12 or 3.13, then it will work on 3.12 and higher. Or you compile with limited API version 3.13 on 3.13, and it won’t load on 3.12 and older.

[rgommers]

This is of course not true. Meson uses importlib.machinery.EXTENSION_SUFFIXES[1]

Thanks for the correction. That makes it more fragile overall; meson-python does do what I described. sysconfig.get_config_var('EXT_SUFFIX') is only a single string value - hard to modify in a backwards compatible way.

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter.

I don't think I've ever seen that in writing. Using a 3.13 interpreter with -DPy_LIMITED_API=<3.12 hex version> in compile flags is common and should work. I'm guessing you meant -DPy_LIMITED_API=<3.13 hex version>?

[eli-schwartz]

I don't think I've ever seen that in writing. Using a 3.13 interpreter with -DPy_LIMITED_API=<3.12 hex version> in compile flags is common and should work. I'm guessing you meant -DPy_LIMITED_API=<3.13 hex version>?

I'm saying that I am pretty sure I've seen CPython core developers say all the time that you may use whatever Py_LIMITED_API you like, and it has the effect of fixing a lower bound, but it doesn't set an upper bound and it doesn't guarantee that the CPython headers won't silently replace functions with macros that use newer ABI than what you expect.

Therefore the only guarantee we have as users is that if you compile an extension using a Py_LIMITED_API specifying version X, it will:

• error out if CPython is X-1
• for CPython >= X, produces a compiled extension that requires at least $CPYTHON_VERSION and pays zero attention to Py_LIMITED_API

Or in simpler terms, specifying a particular version for Py_LIMITED_API does nothing except provide partial linting for cases where your source code is incompatible with the claim that Py_LIMITED_API is making. And lint tools do not an ABI make.

It is unfortunate that no available mechanisms are made use of in the CPython codebase for offering better compile time or runtime guarantees (runtime guarantees for this exact scenario were pioneered decades ago and heavily used by almost every Linux binary on the planet because glibc injects those guarantees whether you know it does or not, every time a compiled binary uses functions from glibc, which is ~100% of them, -+ 0%).