mars-project / mars-project/mars

Reduction over different columns of a single DataFrame can be merged

Open
#2,560 0 comments 0 reactions 0 assignees View on GitHub
mod: dataframe type: enhancement
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

When calculating series aggregations like `execute(df.a.sum(),df.b.mean())`, aggregations over different columns can be merged as `df.agg({'a': 'sum', 'b': 'mean'})`. An optimizer can be added to reduce num of subtasks.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how execute handles separate series aggregations such as df.a.sum() and df.b.mean(), then compare that path with df.agg({'a': 'sum', 'b': 'mean'}). Identify where reduction subtasks are planned and determine whether compatible aggregations can be combined. Done means equivalent results with fewer subtasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.