influxdata / influxdata/influxdb

`build-docker-arm64` CI job broken w/ python library link failure

Open
#26,642 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Here is an example failure:

* https://app.circleci.com/pipelines/github/influxdata/influxdb/47096/workflows/e117d02a-8b5f-4596-8733-81dbaa12114c/jobs/448822

The last successful main branch build can be seen here:

* https://app.circleci.com/pipelines/github/influxdata/influxdb/47093/workflows/33406584-3302-4ba4-966b-7afe855986c3/jobs/448772

The error message looks like this:;

```
#18 37.25 Compiling influxdb3 v3.3.0-nightly (/influxdb3/influxdb3)
#18 837.8 error: linking with `cc` failed: exit status: 1
#18 837.8 |
#18 837.8 = note: "cc" "/tmp/rustcMjbJ27/symbols.o" "<1 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "/tmp/rustcMjbJ27/{libtikv_jemalloc_sys-ad5523c22c9509d7.rlib,libcroaring_sys-6df58cc8aa402811.rlib,liblibsqlite3_sys-822e3d2829f7d4cc.rlib,libblake3-ff06a0b64a50100c.rlib,liblzma_sys-17e82b8b4e6513e0.rlib,libbzip2_sys-891ecf616d46a420.rlib,libpsm-82c8523084128ecc.rlib,libzstd_sys-7dbdbe3b806642cb.rlib,libring-38a3e7093fc0eb5e.rlib}.rlib" "/lib/rustlib/aarch64-unknown-linux-gnu/lib/{libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lpython3.11" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustcMjbJ27/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/influxdb3/target/release/build/ring-4bc31c504a685bc8/out" "-L" "/influxdb3/target/release/build/libsqlite3-sys-006e3a716241d1f3/out" "-L" "/influxdb3/target/release/build/tikv-jemalloc-sys-d2158127411b9215/out/build/lib" "-L" "/influxdb3/target/release/build/bzip2-sys-8cc26a90fc9a23ca/out/lib" "-L" "/influxdb3/target/release/build/lzma-sys-9c3af40cbbb2bb1d/out" "-L" "/influxdb3/target/release/build/zstd-sys-51caaf432f98aecd/out" "-L" "/influxdb3/target/release/build/psm-60a660532246b131/out" "-L" "/influxdb3/target/release/build/blake3-b84ba46ae2356452/out" "-L" "/influxdb3/target/release/build/croaring-sys-16e402e776a77c45/out" "-L" "/influxdb3/python-artifacts/aarch64-unknown-linux-gnu/python/lib" "-L" "/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/influxdb3/target/release/deps/influxdb3-cb5f6fc042a24180" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs"
#18 837.8 = note: some arguments are omitted. use `--verbose` to show all linker arguments
#18 837.8 = note: /usr/bin/ld: cannot find -lpython3.11: No such file or directory
#18 837.8 collect2: error: ld returned 1 exit status
#18 837.8
#18 837.8
#18 837.8 error: could not compile `influxdb3` (bin "influxdb3") due to 1 previous error
#18 ERROR: process "/bin/sh -c du -cshx /usr/local/rustup /usr/local/cargo/registry /usr/local/cargo/git /influxdb3/target && PYO3_CONFIG_FILE=\"/influxdb3/python-artifacts/$PBS_TARGET/pyo3_config_file.txt\" cargo build --target-dir /influxdb3/target --package=\"$PACKAGE\" --profile=\"$PROFILE\" --no-default-features --features=\"$FEATURES\" && objcopy --compress-debug-sections \"target/$PROFILE/$PACKAGE\" && cp \"/influxdb3/target/$PROFILE/$PACKAGE\" \"/root/$PACKAGE\" && patchelf --set-rpath '$ORIGIN/python/lib:$ORIGIN/../lib/influxdb3/python/lib' \"/root/$PACKAGE\" && cp -a \"/influxdb3/python-artifacts/$PBS_TARGET/python\" /root/python && du -cshx /usr/local/rustup /usr/local/cargo/registry /usr/local/cargo/git /influxdb3/target" did not complete successfully: exit code: 101
```

It looks like the job is configured to use Python 3.13:

```
#15 [build 6/9] RUN sed -i "s/^readonly TARGETS=.*/readonly TARGETS=aarch64-unknown-linux-gnu/" ./.circleci/scripts/fetch-python-standalone.bash && ./.circleci/scripts/fetch-python-standalone.bash /influxdb3/python-artifacts "20250612" "3.13.5" && tar -C /influxdb3/python-artifacts -zxf /influxdb3/python-artifacts/all.tar.gz "./aarch64-unknown-linux-gnu" && sed -i 's#tmp/workspace#influxdb3#' "/influxdb3/python-artifacts/aarch64-unknown-linux-gnu/pyo3_config_file.txt" && cat "/influxdb3/python-artifacts/aarch64-unknown-linux-gnu/pyo3_config_file.txt"
#15 CACHED
```

The base image is Debian bookworm, for which the default python is 3.11:

* https://packages.debian.org/search?keywords=python3&searchon=names&exact=1&suite=bookworm§ion=all

I wondered if something in the build tools changed in the past day that would cause it to look for (but be unable to find) the base image's built-in python3 rather than the PBS installation. To test this, earlier today I pushed a commit that only updates the docker base image from the `bookworm` to the `trixie` (soon to be stable) Debian version and the build failure seems to have resolved:

* https://app.circleci.com/pipelines/github/influxdata/influxdb/47106/workflows/72c5d07a-b2bd-4614-8886-dab5fae3347a/jobs/448895

While it seemingly addresses the issue, I don't think that updating from debian `bookworm` to `trixie` is the best fix for this (eventually `trixie`'s python3 will lag behind the PBS version). As far as I can tell, the build process expects the python it links to to be the PBS version, not the system-installed version. I still don't know why that seems to have been the case in the previous main branch build but not the latest 🤔

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the build-docker-arm64 job and .circleci/scripts/fetch-python-standalone.bash, then compare the failed bookworm logs with the successful trixie run. Check the generated pyo3_config_file.txt and the linker's Python library path. Done means the arm64 build links against the fetched Python installation and passes without requiring a Debian base-image upgrade.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, rust
Domain
build-system, ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.