mars-project / mars-project/mars

How to reduce memory cost for concat operation for DataFrame

Open
#2,073 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

同样的问题见 https://stackoverflow.com/questions/56502377/reduce-memory-usage-of-pandas-concat-for-lots-of-dataframes
用pandas拼接多个数据框时会占用大量额外内存。
怎么通过mars或者底层的vineyard避免或者减少这种额外占用?

Contributor guide

Open the contributing guide

Research direction

Start with the linked Stack Overflow question to understand the reported memory cost of concatenating many DataFrames. Then investigate how Mars handles DataFrame concatenation and whether the proposed Mars or Vineyard approach can reduce extra memory usage. Done means a defined approach and evidence that concatenation uses less additional memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.