mars-project / mars-project/mars
[core] speeding up mars tiling
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
For some tileable graph, tile took about 40 minutes:
```
df = o.to_mars_dataframe(input_table_name, columns=["id","a","b"],chunk_size=10000)
df = df.astype({'id': 'string','a':'float','b':'float'})
df.groupby(["id"])["a","b","id"].apply(predictor()).reset_index().execute()
```
`input_table_name` has about 4 billion rows.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
No source files or tests are named. Start by reproducing the provided Mars DataFrame and groupby example, then profile the tiling phase for the four-billion-row input. Done means identifying and implementing a supported way to reduce the roughly 40-minute tiling time, with tests or benchmarks showing the improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100