bazel: python 3.13 header used instead of the one defined by "@rules_python//python/cc:current_py_cc_headers"
- Dominant language
- C++
- Stars
- 14.1k
- Forks
- 2.5k
- Avg merge
- 8h 39m
- Merged PRs (30d)
- 72
Description
TLDR: protobuf use `@system_python` which found my python 3.13 system wide and break the hermiticity of the bazel build.
### Root Cause Analysis
When running:
```
bazel test --config=ci //ortools/math_opt/python:normalized_inequality_test
```
The test fails with:
```
ImportError: .../cpp_elemental.so: undefined symbol: PyDict_GetItemStringRef
```
#### Why did this happen?
1. The missing symbol:
[`PyDict_GetItemStringRef`](https://docs.python.org/3/c-api/dict.html#c.PyDict_GetItemStringRef) was introduced in Python 3.13.0a1 and does not exist in Python 3.12.
2. The dependency chain in ./main:
• Target BUILD.bazel:40-66 depends on `@pybind11_protobuf//pybind11_protobuf:native_proto_caster`.
• `@pybind11_protobuf` depends on `@com_google_protobuf//python:proto_api`.
• In Protobuf 35.1, build_targets.bzl:464-476 depends on `@system_python//:python_headers`.
• Protobuf's MODULE.bazel:138-143 uses system_python_extension which inspects the host system. Because Python 3.13 is installed on this host (/usr/include/python3.13), `@system_python` resolves to:
.../external/protobuf++system_python_extension+system_python/python -> /usr/include/python3.13
3. Include path shadowing:
When Bazel compiles `elemental.cc`, the compiler command line includes:
-isystem external/protobuf++system_python_extension+system_python/python \
...
-isystem external/rules_python++python+python_3_12_x86_64-unknown-linux-gnu/include/python3.12
Because Protobuf's system python path comes before rules_python's hermetic Python 3.12 path, includes from /usr/include/python3.13/Python.h.
4. pybind11 3.0.1 code generation:
In pybind11 3.0.1 (pinned in main), internals.h:570 calls dict_getitemstringref(). In pytypes.h:1001, dict_getitemstringref() checks:
#if PY_VERSION_HEX >= 0x030D00A1
if (PyDict_GetItemStringRef(v, key, &rv) < 0) { ... }
Since PY_VERSION_HEX was evaluated from Python 3.13 headers, a call to PyDict_GetItemStringRef was emitted into elemental.pic.o.
5. Runtime execution:
normalized_inequality_test is run using rules_python hermetic Python 3.12 runtime. Loading cpp_elemental.so fails immediately because libpython3.12.so does not define PyDict_GetItemStringRef.
────
### Why this cannot be reproduced in pybind11_bazel/examples/basic
1. No Protobuf / @system_python dependency:
pybind11_bazel only depends on @rules_python//python/cc:current_py_cc_headers. In basic, basic.cpp compiles exclusively against rules_python hermetic headers (3.11 by default, or 3.12 when configured). It never includes /usr/include/python3.13.
2. basic.cpp sees PY_VERSION_HEX < 0x030D0000:
Even if basic.cpp includes pybind11/pytypes.h and calls dict_getitemstringref, the preprocessor condition #if PY_VERSION_HEX >= 0x030D00A1 evaluates to false, selecting dict_getitemstring (PyDict_GetItemWithError) instead of PyDict_GetItemStringRef.
3. pybind11 3.0.3 removed the call:
pybind11_bazel uses pybind11 3.0.3, which already refactored internals.h to avoid calling dict_getitemstringref altogether.
Contributor guide
Assessment
This issue has not been assessed yet.