mars-project / mars-project/mars
Reduction over different columns of a single DataFrame can be merged
Open
mod: dataframe
type: enhancement
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
When calculating series aggregations like `execute(df.a.sum(),df.b.mean())`, aggregations over different columns can be merged as `df.agg({'a': 'sum', 'b': 'mean'})`. An optimizer can be added to reduce num of subtasks.
Contributor guide
Research direction
Start by tracing how execute handles separate series aggregations such as df.a.sum() and df.b.mean(), then compare that path with df.agg({'a': 'sum', 'b': 'mean'}). Identify where reduction subtasks are planned and determine whether compatible aggregations can be combined. Done means equivalent results with fewer subtasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100