pypa / pypa/setuptools

[FR] extensions: general way to provide library paths that are searched before `LDSHARED`'s paths

Open
#4,843 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Needs Triage
Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

What's the problem this feature will solve?

Currently, if sysconfig's LDSHARED contains a -L flag, that flag will appear before user-provided flags when building an extension. If the user provides an additional path to search (e.g. via LDFLAGS), that path will have lower priority than paths in LDSHARED since library paths are searched left to right. This makes it hard for a user to prefer their own library, even when an undesired version is being found via LDSHARED.

Describe the solution you'd like

Ideally, there would be a way to search user-provided paths first. But I'm honestly not sure what this would look like, implementation-wise. Since LDSHARED contains both the linker invocation and flags to the linker, any solution would seem to need to parse LDSHARED to insert -L flags early. That seems fragile. However, there is precedent in the sense that LDSHARED is already automatically rewritten with a user-provided CC: https://github.com/pypa/setuptools/blob/ba243756233d0afe944db6e02ddcb70064dcd22c/setuptools/_distutils/sysconfig.py#L331

Alternative Solutions

Users can currently pass LDSHARED to fully specify the linker command and the order of -L flags. This is currently the workaround I'm using. However, it's verbose (see below) and arguably error-prone, since it requires manually discovering the LDSHARED value and modifying it to insert your own paths first.

Additional context

Here's an example LDSHARED on my system, which contains a lot of paths a user might want to override (notably -L/usr/lib64):

> python -m sysconfig
...
LDSHARED = "/mnt/sw/nix/store/qgcgvdlgz6546x6mzjiknvrq218j9nx8-gcc-11.4.0/bin/gcc -pthread -shared -L/mnt/sw/nix/store/30zwhjaf2jh84r0bws1yz2dwgqhw23f0-bzip2-1.0.8/lib -L/mnt/sw/nix/store/q0d0y49rsz7291c00kr0560y2q6bix58-expat-2.6.2/lib -L/mnt/sw/nix/store/snfa89v7a38qmzwqsj7006awc2z77rf4-gdbm-1.23/lib -L/mnt/sw/nix/store/zdnba5ggf37q6cxx1rszxabi8m260cfh-gettext-0.22.4/lib -L/mnt/sw/nix/store/d9n53v93vxra9a0madzpkicmrxxgbd61-libffi-3.4.6/lib64 -L/mnt/sw/nix/store/6416cbjlqkw7l9q7naz81ha4mknw3pkr-libxcrypt-4.4.35/lib -L/mnt/sw/nix/store/hn8dq06sw3hyx5z21z1n2kf62d37n8sv-ncurses-6.4/lib -L/usr/lib64 -L/mnt/sw/nix/store/nfr8l146l2lfxhdphgmd63qb8awy07w7-readline-8.2/lib -L/mnt/sw/nix/store/i117x95p5jmjpqmkdp8iyvr6n4yhzn6i-sqlite-3.43.2/lib -L/mnt/sw/nix/store/0swn26vgd2lwfnindbxcgzbpz2ra2qmc-tcl-8.6.12/lib -L/mnt/sw/nix/store/wl8bvjd4iwsx496lbpy641ihpswldqry-tk-8.6.11/lib -L/mnt/sw/nix/store/rrkjvd4n6dg3y0wvr2r34zq6zfz7864a-util-linux-uuid-2.38.1/lib -L/mnt/sw/nix/store/pmzfq3af88pyl5ykcrxqjqa1k4639nnl-xz-5.4.6/lib -L/mnt/sw/nix/store/2i17p7mws5s8x6y68md7vyclaim4b7nk-zlib-1.3.1/lib"
...
Code of Conduct
  • I agree to follow the PSF Code of Conduct

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in setuptools/_distutils/sysconfig.py around the existing LDSHARED rewrite for a user-provided CC, then trace how LDSHARED and LDFLAGS reach extension builds. Define a way for user-provided library paths to precede LDSHARED paths without requiring a complete LDSHARED override; done means the ordering is supported without fragile manual command parsing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.