apache / apache/arrow

[pyarrow] significant increase in memory when connect hdfs and upload files to hdfs

Open
#14,268 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

pyarrow version:7.0.0
connect hdfs to get a fileSystem object by pyarrow,the memory increased from 64M to 254M,
then I use this fileSystem object to upload a 2G file to hdfs, the memory increased to 289M
after 10 min, the memory keep the same.
how can i release the memory?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reproducing the report with pyarrow 7.0.0: connect to HDFS, upload a 2G file, and measure memory before connection, after connection, after upload, and after 10 minutes; done means determining whether the increase is expected and whether the memory can be released.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.