pybind / pybind/pybind11

[BUG]: Python3.11 `error_already_set.what()` segfault

Open
#4,853 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Required prerequisites
What version (or hash if on master) of pybind11 are you using?

2.11.1

Problem description

I run an import of pyarrow and attempt to create a Table from a Pandas DataFrame

return py::module_::import("pyarrow").attr("lib").attr("Table").attr("from_pandas")(df);

this throws an exception, which I catch:

   103          } catch (py::error_already_set &e) {
-> 104                  throw InvalidInputException(
   105                      "The dataframe could not be converted to a pyarrow.lib.Table, due to the following python exception: %s",
   106                      e.what());
   107          }

py::error_already_set.what() causes the segfault.

Reproducible example code

I maintain an opensource repo and I absolutely hate it when I get an issue without a reproduction example, but in this case I can't really make one as it's wrapped into an application and only occurs when run together with multiple tests.

(Just running the test where the segfault occurs does not reproduce the issue)

I have however attached lldb and have a stacktrace that hopefully provides more information:

* thread #1, name = 'python3', stop reason = signal SIGSEGV: invalid address (fault address: 0xa8)
    frame #0: 0x00000000005a4e6c python3`PyFrame_GetBack + 44
python3`PyFrame_GetBack:
->  0x5a4e6c <+44>: ldrsw  x6, [x4, #0xa8]
    0x5a4e70 <+48>: add    x7, x3, x6, lsl #1
    0x5a4e74 <+52>: cmp    x5, x7
    0x5a4e78 <+56>: b.lo   0x5a4ecc                  ; <+140>
(lldb) up
error: duckdb.cpython-311-aarch64-linux-gnu.so 0x017e4a81: DW_TAG_member '_M_pod_data' refers to type 0x01912765 which extends beyond the bounds of 0x017e4a2c
frame #1: 0x0000fffff3892fa8 duckdb.cpython-311-aarch64-linux-gnu.so`pybind11::detail::error_fetch_and_normalize::format_value_and_trace[abi:cxx11](this=0x000000000262fd70) const at pytypes.h:657:48
   654                  result += '\n';
   655                  Py_DECREF(f_code);
   656  #    if PY_VERSION_HEX >= 0x030900B1
-> 657                  auto *b_frame = PyFrame_GetBack(frame);
   658  #    else
   659                  auto *b_frame = frame->f_back;
   660                  Py_XINCREF(b_frame);
(lldb) up
frame #2: 0x0000fffff3893298 duckdb.cpython-311-aarch64-linux-gnu.so`pybind11::detail::error_fetch_and_normalize::error_string[abi:cxx11](this=0x000000000262fd70) const at pytypes.h:682:65
   679 
   680      std::string const &error_string() const {
   681          if (!m_lazy_error_string_completed) {
-> 682              m_lazy_error_string += ": " + format_value_and_trace();
   683              m_lazy_error_string_completed = true;
   684          }
   685          return m_lazy_error_string;
(lldb) 
frame #3: 0x0000fffff38a7b00 duckdb.cpython-311-aarch64-linux-gnu.so`pybind11::error_already_set::what(this=0x000000000246c690) const at pybind11.h:2668:41
   2665 inline const char *error_already_set::what() const noexcept {
   2666     gil_scoped_acquire gil;
   2667     error_scope scope;
-> 2668     return m_fetched_error->error_string().c_str();
   2669 }
   2670
   2671 PYBIND11_NAMESPACE_BEGIN(detail)
(lldb) 
frame #4: 0x0000fffff39b2c14 duckdb.cpython-311-aarch64-linux-gnu.so`duckdb::ArrowTableFromDataframe(df=0x0000ffffffffb718) at pyconnection.cpp:104:3
   101          try {
   102                  return py::module_::import("pyarrow").attr("lib").attr("Table").attr("from_pandas")(df);
   103          } catch (py::error_already_set &e) {
-> 104                  throw InvalidInputException(
   105                      "The dataframe could not be converted to a pyarrow.lib.Table, due to the following python exception: %s",
   106                      e.what());
   107          }

Python version: Python 3.11.0rc1
(Docker) Linux version: Linux f63acc518afb 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

On Python3.9 the issue is not reproducable



### Is this a regression? Put the last known working version here if it is.

Not a regression

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing pybind11.h around error_already_set::what() and pytypes.h around error_fetch_and_normalize::format_value_and_trace(), then try the reported pyarrow/Pandas path under Python 3.11. Compare the failing Python 3.11 behavior with Python 3.9 and the provided LLDB stack trace. Done means the exception can be inspected without a segmentation fault and the behavior is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.