apache / apache/arrow

[C++] Potential memory leak at shutdown if an exec plan with a scanner fails or is aborted immediately before shutdown

Open
#20,338 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I'm primarily creating this so we can remember to make a test for this. This problem should be solved as part of ARROW-16072. When the scanner fails it simply discards references to the various scanner AsyncGenerators. However, some I/O tasks may still have references to these generators and so some parts of the scanner survive after the plan itself is marked complete. If there is an immediate shutdown then these parts will not be properly disposed of even though the plan is marked complete and it will show up as a memory leak.

Example:

https://pipelines.actions.githubusercontent.com/serviceHosts/8bb0d999-3387-4c48-9fa6-c66c718a46e2/_apis/pipelines/1/runs/359690/signedlogcontent/4?urlExpires=2022-07-25T14%3A43%3A01.2797488Z&urlSigningMethod=HMACV1&urlSignature=GS3lS09Q9sTRweN%2B8UEu2GwUGc%2FbO9eyH27FRKumbrg%3D

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-17198) / @westonpace

**Note**: *This issue was originally created as [ARROW-17198](https://issues.apache.org/jira/browse/ARROW-17198). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

No source file or test path is identified. Start by reviewing ARROW-16072 and the scanner/exec-plan shutdown behavior it covers; done means adding a regression test for a failed or aborted scanner immediately before shutdown and verifying that no memory leak remains.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.