apache / apache/incubator-resilientdb

[BUG] pybind11 2.6.2 does not build against Python 3.11+, so `bazel build ...` fails on current Ubuntu

Open
#262 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
169
Forks
277
Avg merge
18h 51m
Merged PRs (30d)
1

Description

## Summary

`WORKSPACE` pins pybind11 2.6.2, which predates Python 3.11 making
`PyFrameObject` an opaque type. Ubuntu 24.04 ships Python 3.12 and 26.04 ships
3.14, so every target that compiles source against the pybind11 2.6.2 headers
fails there, and with them `bazel build ...`.

## Reproduction

```
docker run --rm -it ubuntu:24.04 bash
apt-get update && apt-get install -y sudo curl wget gnupg ca-certificates git
git clone https://github.com/apache/incubator-resilientdb && cd incubator-resilientdb
./INSTALL.sh
bazel build //api:pybind_kv.so
```

## Result

On Ubuntu 24.04.4, Python 3.12.3, gcc 13:

```
ERROR: api/BUILD:21:10: Compiling api/pybind_kv_service.cpp failed
external/pybind11/include/pybind11/cast.h:446:36: error: invalid use of
incomplete type 'PyFrameObject' {aka 'struct _frame'}
external/pybind11/include/pybind11/cast.h:446:75: error: expected
primary-expression before '>' token
external/pybind11/include/pybind11/cast.h:448:29: error: invalid use of
incomplete type 'PyFrameObject' {aka 'struct _frame'}
```

A separate full-tree run on 24.04 also reported `PyThreadState` having no
member `frame`, from `pybind11.h` rather than `cast.h`; that member stopped
being part of the public structure in Python 3.11, so it is the same
incompatibility surfacing elsewhere. That run also reported `function_record`
having no member `nargs_pos_only`, which may be a cascade from the missing
`` under GCC 13 rather than independent evidence about the Python
version, so I am not relying on it here.

Ubuntu 26.04 ships Python 3.14, which is further still from what 2.6.2
supports, but I have not run this target there.

`//service/kv:kv_service` built successfully in the same container in the same
run, so this is specific to the pybind11 dependency rather than a general build
failure.

## Affected targets

- `//api:pybind_kv.so` (`api/BUILD:22`) — verified failing
- `//service/tools/utxo/wallet_tool/pybind:...`
- `//common:pybind` (`common/BUILD:58`) is a deps-only aggregator with no
`srcs`; anything that compiles through it is affected

(Line numbers as of master at the time of writing.)

Because these are in the tree, `bazel build ...` cannot succeed on Ubuntu 24.04
or newer.

## Why this is not visible in CI

`build.yml:45` and `ut.yml:45` both pin `python-version: '3.10'`, which is what
Ubuntu 22.04 ships. The pinned Python is older than the version at which
`PyFrameObject` became opaque, so the failure cannot appear there.

## Notes

`WORKSPACE:236` pins `pybind11_bazel` 2.11.1.bzl.1 while `WORKSPACE:244` pins
pybind11 itself at 2.6.2, so the two are already several years apart. (Line
numbers are as of master at the time of writing.)

pybind11 added Python 3.11 support in the 2.10 series and Python 3.13 in 2.13;
Python 3.14 needs pybind11 3.x, which is a major release with API changes. The
right target version therefore depends on which Ubuntu releases the project
wants to support, which is why this is filed rather than fixed here.

Found while working on issue #245; the pull request that fixes it builds
`//service/kv:kv_service` rather than `...` for this reason.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with WORKSPACE:236 and WORKSPACE:244 to compare the pybind11_bazel and pybind11 pins, then inspect api/BUILD:22 and the affected pybind targets. Reproduce with Python 3.12 using `bazel build //api:pybind_kv.so`, and check build.yml:45 and ut.yml:45 for CI coverage. Done means the affected targets build on the supported Ubuntu/Python versions without the reported pybind11 errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, ubuntu
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.