apache / apache/arrow

[Python] Multiple warnings and asserts triggered in debug CPython 3.11

Open
#33,765 9 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

CPython [can be built in debug mode](https://docs.python.org/3/using/configure.html#python-debug-build) to catch some maybe-fatal-maybe-not errors. We have such python3.11 configured with `--with-pydebug` and pyarrow 10.0.1 installed, here is an example of gc warning
```
>>> import pyarrow as pa
>>> table = pa.table({'a': [1]})
gc:0: ResourceWarning: Object of type pyarrow.lib.Int64Array is not untracked before destruction
```
similar code sometimes triggers assertion
```
gc:0: ResourceWarning: Object of type pyarrow.lib.UInt16Array is not untracked before destruction
Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address : 0x7f804e8762e0
object refcount : 0
object type : 0x7f80fcc3f5e0
object type name: pyarrow.lib.UInt16Array
object repr :

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
```
Another crash
```
>>> import pyarrow as pa
>>> pa.table({0: []})
python: Objects/typeobject.c:1068: type_call: Assertion `!_PyErr_Occurred(tstate)' failed.
Aborted (core dumped)
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the warnings and crashes with a debug CPython 3.11 build, pyarrow 10.0.1, and the examples in the issue. Inspect the reported paths in Modules/gcmodule.c and Objects/typeobject.c, then trace the pyarrow objects involved; done means the examples no longer emit the warnings, assertions, or fatal errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.