apache / apache/arrow

[Python] Possible memory leak in compute.take

Open
#31,456 7 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

If you run the following code, the memory usage of the process goes up to 1GB even though the pyarrow allocated bytes is always at ~80MB. The process memory comes down after a while to 800 MB, but is still way more than what is necessary.

'''

import pyarrow as pa
import numpy as np
import pandas as pd
import os, psutil
import pyarrow.compute as compute
import gc
my_table = pa.Table.from_pandas(pd.DataFrame(np.random.normal(size=(10000,1000))))

process = psutil.Process(os.getpid())
print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())

for i in range(100):
    print("mem usage", process.memory_info().rss, pa.total_allocated_bytes())
    temp = compute.sort_indices(my_table['0'], sort_keys = [('0','ascending')])
    my_table = my_table.take(temp)
    gc.collect()

'''

**Environment**: Ubuntu
**Reporter**: [Ziheng Wang](https://issues.apache.org/jira/browse/ARROW-16037) / @marsupialtail

**Note**: *This issue was originally created as [ARROW-16037](https://issues.apache.org/jira/browse/ARROW-16037). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

First reproduce the reported script on Ubuntu and compare process RSS with pyarrow's allocated-byte count while exercising compute.sort_indices and Table.take. Trace those entry points to identify where retained memory is handled; done means repeated iterations no longer cause unexplained RSS growth beyond the reported allocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.