apache / apache/arrow

[C++] [Acero] Use-after-free of a MemoryPool that was destroyed at program's exit

Open
#35,959 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I don't have a small reproducer but looking at the stack traces and the code I
have I'm pretty sure that I observe a bug despite using the API correctly.

I set up an ExecPlan and dispose of it, via StopProducing()+finished.Wait() calls.
Only default context (memory pool, executors, etc.) is ever used.
Shortly after, the program finishes, and the destructor of the default memory pool is called.
However, in another thread the wheels are still spinning somewhere in the executor code.
To be a bit more precise, a Future is being executed that I did not submit and that I believe
is an internal implementation detal. The execution of this Future asks for some memory
but the pool has been destroyed by then.

I'd like to know whether what I've described is a known issue and what the suggested
(client-side) fix is.

These previous issues seem relevant:
https://issues.apache.org/jira/browse/ARROW-16072
https://issues.apache.org/jira/browse/ARROW-17198

The mitigation in https://github.com/apache/arrow/pull/13691 is good enough for tests but not for production
use (consider the case of more than one instance of Scanner).

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the default memory-pool and executor lifecycle described in the issue, then compare ARROW-16072, ARROW-17198, and PR #13691. Done means establishing the lifecycle race and documenting or implementing a safe resolution that remains correct with multiple Scanner instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.