mars-project / mars-project/mars
How to reduce memory cost for concat operation for DataFrame
Open
question
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
同样的问题见 https://stackoverflow.com/questions/56502377/reduce-memory-usage-of-pandas-concat-for-lots-of-dataframes
用pandas拼接多个数据框时会占用大量额外内存。
怎么通过mars或者底层的vineyard避免或者减少这种额外占用?
Contributor guide
Research direction
Start with the linked Stack Overflow question to understand the reported memory cost of concatenating many DataFrames. Then investigate how Mars handles DataFrame concatenation and whether the proposed Mars or Vineyard approach can reduce extra memory usage. Done means a defined approach and evidence that concatenation uses less additional memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100