pyarrow.csv.read_csv() memory not released in jupyterlab
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
The memory not released after execute load_csv_by_arrow() function on JupyterLab. This seems only happen in JupyterLab not other IDE and the memory seems increase a little bit every time the function runs.
```
def load_csv_by_arrow():
table = csv.read_csv(r'./data/xxx.csv)
df = table.to_pandas()
del df
# in JupyterLab cell
%memit load_csv_by_arrow()
```
pyarrow:11.0.0
jupyterlab: 3.6.3
### Component(s)
Python
Contributor guide
Research direction
Reproduce the repeated load_csv_by_arrow() call in a JupyterLab cell using pyarrow.csv.read_csv() and table.to_pandas(), then compare memory behavior across runs. Trace the Python CSV-reading and pandas-conversion entry points to identify where retained memory is associated with the JupyterLab case. Done means the demonstrated workload no longer shows steadily increasing retained memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100