musl-cross-make cross compiling and specifying "--with-libc=/custom/path/to/libc.so" is dismissed

Open
#97,842 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python

Research direction

Start with the build recipe in modules/python and the generated config.log, then trace how configure handles --with-libc during the cross-build path. Compare the documented OpenWrt Python patches and the CircleCI failure to identify where the host libc is selected. Done means the specified cross-built libc is used instead of /usr/lib/x86_64-linux-gnu/libc.so.

Written by the indexing model from the issue text.

Description

build type-bug

Bug report

Trying to integrate CPython 3.9.2 under Heads firmware (coreboot+linux payload). This is followup of old integration issue at https://github.com/osresearch/heads/issues/689

CPython 3.9.2 (as all other modules under Heads but coreboot) is built by first building musl-cross-make and then having modules being build with musl-cross-make to have reproducible build environment, so that coreboot payload' linux kernel's initrd contains binaries needed in linux embedded environment (recovery shell and associated tools), are reproducible (this is the goal).

Your environment

  • CPython versions tested on: 3.9.2 installed to build target 3.9.2 python
  • Operating system and architecture: debian-11 host build environement

I'm experimenting with the following build recipe, and did a lot of variation without success https://github.com/osresearch/heads/blob/df50c5f7d242b502fc443157fd592b638da87af0/modules/python#L13-L33.

This results in configure script producing the following config.log output:
config.log

So above build recipe produces:

  • a correct ./configure callm specifying ./configure --enable-optimizations --with-cxx-main=/home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-gcc --with-libc=/home/user/heads/crossgcc/x86/x86_64-linux-musl/lib/libc.so --host x86_64-linux-musl --build x86_64 --disable-ipv6 --prefix= --exec-prefix=
  • --with-libc="$(musl-cross_libraries)" translates into config.log, correctly, into configure:14435: result: set LIBC="/home/user/heads/crossgcc/x86/x86_64-linux-musl/lib/libc.so". This is the libc.so we need to rely on
  • Unfortunately, the build fails (See in CircleCI logs here) with libc being completely dismissed. This results into error : ImportError: /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header. This is the wrong libc.so of the host, not of the cross built environment.

Expected behavior:

  • Specifying --with-libc=/home/user/heads/crossgcc/x86/x86_64-linux-musl/lib/libc.so should result in libc.so used from /home/user/heads/crossgcc/x86/x86_64-linux-musl/lib/libc.so not /usr/lib/x86_64-linux-gnu/libc.so.

Why is the host libc used even when libc is specified and found at configure?


Was gonna investigate and adapt OpenWrt patches at https://github.com/openwrt/packages/tree/master/lang/python/python3/patches but reading about cross build issues it would be better to have this fixed upstream then downstream into all projects trying to cross build for their embedded systems.

Any insight?

Dominant language
Python
Stars
77.2k
Forks
36k
Avg merge
1d 9h
Merged PRs (30d)
558

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/cpython

All issues in python/cpython

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.