[C++] Weighted stat aggregations in arrow-compute
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
Arrow’s compute functions currently include several aggregate statistics (mean, sum, variance, etc).
It would be great to offer weighted versions of several of these (mean, sum, count, variance and standard deviation in short run, quantile in longer-run) and expose them to pyarrow, R, etc. This would allow this functionality to be pushed down to Arrow and Arrow datasets. For example, an R user would have to either collect the data first and apply something like weighted.mean from R stats or code up custom logic using functionality available in Arrow to arrive at a similar result. Eother is not ideal for a relatively routine aggregation (weighted statistics). It would also make it much easier to calculate weighted statistics when working with distributed Arrow datasets in Python.
Note that this is not covered by UDF API directly which doesn’t support aggregate functions.
The added functionality could be through the creation of functions with a signature like: weighted_mean(x: arrow array, weights: arrow array) -> scalar
### Component(s)
C++, Python, R
Contributor guide
Research direction
Start by reviewing Arrow’s existing aggregate statistics in the C++ compute component and how they are exposed to Python and R. Define a focused initial scope from the requested weighted mean, sum, count, variance, and standard deviation, then identify corresponding tests and bindings; done means the selected functions work across the intended interfaces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python, r
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100