apache / apache/arrow

[Python][C++] Simplify/Validate PyObject refcount in udf.cc

Open
#36,000 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

In udf.cc, there is interaction between C++ and Python where the C++ code holds a user defined Python object inside a Kernel and invoke it over input data.

Due to this interaction, there are a number of `Py_INCREF` call as well as using `OwnRef` to manage the refcount for the PyObject (which represents the user defined function. The current code for managing it is either over complicated and/or bug where the C++ code might be over increasing the refcount. While this is not a huge issue (it probably cause the Python function to never be garbage collected), it could use some clean up / documentation as least to improve maintainability.

Appendix:
* Related thread: https://github.com/apache/arrow/pull/35514#discussion_r1205081272

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Start by reading udf.cc and the related discussion in apache/arrow#35514, tracing each Py_INCREF and OwnRef use around the user-defined function held by the Kernel. Done means the reference-count handling is simpler, documented where needed, and no longer over-increments the PyObject.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.