[C++] Proxy memory pool crashes with Dataset scanning
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Discovered while trying to use the proxy memory pool for testing ARROW-18164
See https://github.com/apache/arrow/pull/14516#discussion_r1005433867
This test segfaults (using the fixture in `test_dataset.py`:
```python
@pytest.mark.parquet
def test_scanner_proxy_memory_pool(dataset):
proxy_pool = pa.proxy_memory_pool(pa.default_memory_pool())
_ = dataset.to_table(memory_pool=proxy_pool)
```
Response of @westonpace:
> My guess is that the problem is that the scanner erroneously returns before all work is completely finished. Changing the thread pool or the memory pool too quickly after a scan can lead to this kind of error. The new scanner was created specifically to avoid this problem but it isn't the default yet (still working through some follow-up PRs to make sure we have the same functionality).
So once that becomes the default new scanner, we can see if this is fixed.
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-18293) / @jorisvandenbossche
**Note**: *This issue was originally created as [ARROW-18293](https://issues.apache.org/jira/browse/ARROW-18293). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce test_scanner_proxy_memory_pool in test_dataset.py using the proxy memory pool and dataset fixture. Read the scanner implementation and the linked discussion about work finishing after a scan; done means the test completes without a segmentation fault when the proxy pool is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100