TDigest quantile estimate algorithm correctness
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
> BTW, the estimate quantile algorithm doesn't follow the `paper`, any reason for this?
>
> https://github.com/apache/arrow-datafusion/blob/df8aa7a2e2a6f54acfbfed336b84144256fb7ff8/datafusion/physical-expr/src/aggregate/tdigest.rs#L523-L524
>
>
>
> 
>
>
_Originally posted by @HaoYang670 in [#4259](https://github.com/apache/datafusion/issues/4259#issuecomment-1325995968)_
Contributor guide
Research direction
Start with datafusion/physical-expr/src/aggregate/tdigest.rs around lines 523-524 and compare the quantile estimate logic with the linked paper. Determine the expected algorithm and the specific discrepancy before changing anything; done means the implementation matches the paper and the behavior is validated against the relevant DataFusion aggregation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100