mars-project / mars-project/mars

[core] speeding up mars tiling

Open
#2,866 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

For some tileable graph, tile took about 40 minutes:
```
df = o.to_mars_dataframe(input_table_name, columns=["id","a","b"],chunk_size=10000)
df = df.astype({'id': 'string','a':'float','b':'float'})
df.groupby(["id"])["a","b","id"].apply(predictor()).reset_index().execute()
```
`input_table_name` has about 4 billion rows.
![image](https://user-images.githubusercontent.com/12445254/160046337-56b57551-c4bc-4cf7-9ab6-2cfe49abf43b.png)

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the provided Mars DataFrame and groupby example, then profile the tiling phase for the four-billion-row input. Done means identifying and implementing a supported way to reduce the roughly 40-minute tiling time, with tests or benchmarks showing the improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.