[C++] [Acero] Use-after-free of a MemoryPool that was destroyed at program's exit
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
I don't have a small reproducer but looking at the stack traces and the code I
have I'm pretty sure that I observe a bug despite using the API correctly.
I set up an ExecPlan and dispose of it, via StopProducing()+finished.Wait() calls.
Only default context (memory pool, executors, etc.) is ever used.
Shortly after, the program finishes, and the destructor of the default memory pool is called.
However, in another thread the wheels are still spinning somewhere in the executor code.
To be a bit more precise, a Future is being executed that I did not submit and that I believe
is an internal implementation detal. The execution of this Future asks for some memory
but the pool has been destroyed by then.
I'd like to know whether what I've described is a known issue and what the suggested
(client-side) fix is.
These previous issues seem relevant:
https://issues.apache.org/jira/browse/ARROW-16072
https://issues.apache.org/jira/browse/ARROW-17198
The mitigation in https://github.com/apache/arrow/pull/13691 is good enough for tests but not for production
use (consider the case of more than one instance of Scanner).
### Component(s)
C++
Contributor guide
Research direction
Start by reviewing the default memory-pool and executor lifecycle described in the issue, then compare ARROW-16072, ARROW-17198, and PR #13691. Done means establishing the lifecycle race and documenting or implementing a safe resolution that remains correct with multiple Scanner instances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100