[Python] Possible memory leak in compute.take
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
If you run the following code, the memory usage of the process goes up to 1GB even though the pyarrow allocated bytes is always at ~80MB. The process memory comes down after a while to 800 MB, but is still way more than what is necessary.
'''
import pyarrow as pa
import numpy as np
import pandas as pd
import os, psutil
import pyarrow.compute as compute
import gc
my_table = pa.Table.from_pandas(pd.DataFrame(np.random.normal(size=(10000,1000))))
process = psutil.Process(os.getpid())
print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())
for i in range(100):
print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())
temp = compute.sort_indices(my_table['0'], sort_keys = [('0','ascending')])
my_table = my_table.take(temp)
gc.collect()
'''
**Environment**: Ubuntu
**Reporter**: [Ziheng Wang](https://issues.apache.org/jira/browse/ARROW-16037) / @marsupialtail
**Note**: *This issue was originally created as [ARROW-16037](https://issues.apache.org/jira/browse/ARROW-16037). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
First reproduce the reported script on Ubuntu and compare process RSS with pyarrow's allocated-byte count while exercising compute.sort_indices and Table.take. Trace those entry points to identify where retained memory is handled; done means repeated iterations no longer cause unexplained RSS growth beyond the reported allocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100