apache / apache/datafusion

Statistics: Implement `SampledDistribution` variant to `Distribution` to support estimated distributions

Open
#14,897 3 comments 0 reactions 1 assignee Claimed by @cj-zhukov View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The new low-level framework to evaluate expressions in a probabilistic context (#14699) supports various distribution types, but it doesn't yet support sampled distributions represented by historgrams.

### Describe the solution you'd like

We can add a `SampledDistribution` variant to `Distribution`, which will store a histogram. All summary statistics (mean, variance and others) will be derived from the histogram. When combining `SampledDistribution` objects with binary operations, the statistics framework will construct the resulting histogram from histograms of operands. For example, when `evaluate_statistics` is invoked on a `BinaryExpr` that *adds* two expressions whose `Distributions` are `SampledDistribution`s, the output histogram will be the convolution of the two histograms (due to the independence assumption in the framework).

### Describe alternatives you've considered

N/A.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.