[Python] Memory leak when using engine="pyarrow" while reading csv files via pd.read_csv()

Open

#44,147 opened on Sep 17, 2024 0 comments 0 reactions 0 assignees View on GitHub

Component: Other Component: Python Type: bug

Dominant language: C++
Stars: 17.1k
Forks: 4.3k
Avg merge: 3d 13h
Merged PRs (30d): 88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I have a 3.5GB csv file and I'm using pyarrow as the engine to speed up the read time. Unfortunately it causes memory to be allocated and not freed afterwards.

![image_2024-09-17_134817340](https://github.com/user-attachments/assets/c35ce3e1-a7f9-4c0c-a7d8-26a24e07093d)
![image](https://github.com/user-attachments/assets/77531358-00b9-4cd4-bb54-8a854eb579bc)

![image](https://github.com/user-attachments/assets/85c11732-8daf-47f3-b6ea-8107ed15f8b3)

OS: windows 10
CPU: i9-12900H

### Component(s)

Python, Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

[Python] Memory leak when using engine="pyarrow" while reading csv files via pd.read_csv()

Description

Contributor guide

Assessment

Get new issues in your inbox