apache / apache/arrow

[C++] Proxy memory pool crashes with Dataset scanning

Open
#33,470 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Discovered while trying to use the proxy memory pool for testing ARROW-18164

See https://github.com/apache/arrow/pull/14516#discussion_r1005433867

This test segfaults (using the fixture in `test_dataset.py`:

```python

@pytest.mark.parquet
def test_scanner_proxy_memory_pool(dataset):
proxy_pool = pa.proxy_memory_pool(pa.default_memory_pool())
_ = dataset.to_table(memory_pool=proxy_pool)
```

Response of @westonpace:

> My guess is that the problem is that the scanner erroneously returns before all work is completely finished. Changing the thread pool or the memory pool too quickly after a scan can lead to this kind of error. The new scanner was created specifically to avoid this problem but it isn't the default yet (still working through some follow-up PRs to make sure we have the same functionality).

So once that becomes the default new scanner, we can see if this is fixed.

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-18293) / @jorisvandenbossche

**Note**: *This issue was originally created as [ARROW-18293](https://issues.apache.org/jira/browse/ARROW-18293). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce test_scanner_proxy_memory_pool in test_dataset.py using the proxy memory pool and dataset fixture. Read the scanner implementation and the linked discussion about work finishing after a scan; done means the test completes without a segmentation fault when the proxy pool is used.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.