apache / apache/arrow

pyarrow.csv.read_csv() memory not released in jupyterlab

Open
#34,934 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The memory not released after execute load_csv_by_arrow() function on JupyterLab. This seems only happen in JupyterLab not other IDE and the memory seems increase a little bit every time the function runs.
```
def load_csv_by_arrow():
table = csv.read_csv(r'./data/xxx.csv)
df = table.to_pandas()
del df

# in JupyterLab cell
%memit load_csv_by_arrow()
```
pyarrow:11.0.0
jupyterlab: 3.6.3

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the repeated load_csv_by_arrow() call in a JupyterLab cell using pyarrow.csv.read_csv() and table.to_pandas(), then compare memory behavior across runs. Trace the Python CSV-reading and pandas-conversion entry points to identify where retained memory is associated with the JupyterLab case. Done means the demonstrated workload no longer shows steadily increasing retained memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.