apache / apache/arrow

[C++] Weighted stat aggregations in arrow-compute

Open
#15,103 8 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Component: R Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

Arrow’s compute functions currently include several aggregate statistics (mean, sum, variance, etc).

It would be great to offer weighted versions of several of these (mean, sum, count, variance and standard deviation in short run, quantile in longer-run) and expose them to pyarrow, R, etc. This would allow this functionality to be pushed down to Arrow and Arrow datasets. For example, an R user would have to either collect the data first and apply something like weighted.mean from R stats or code up custom logic using functionality available in Arrow to arrive at a similar result. Eother is not ideal for a relatively routine aggregation (weighted statistics). It would also make it much easier to calculate weighted statistics when working with distributed Arrow datasets in Python.

Note that this is not covered by UDF API directly which doesn’t support aggregate functions.

The added functionality could be through the creation of functions with a signature like: weighted_mean(x: arrow array, weights: arrow array) -> scalar

### Component(s)

C++, Python, R

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Arrow’s existing aggregate statistics in the C++ compute component and how they are exposed to Python and R. Define a focused initial scope from the requested weighted mean, sum, count, variance, and standard deviation, then identify corresponding tests and bindings; done means the selected functions work across the intended interfaces.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, r
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.