Refactor covariance function to numerical stable version
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
As @snleee has pointed out, the current implementation for covar is not numerical stable and may subject to https://en.wikipedia.org/wiki/Catastrophic_cancellation if the covariance value itself is small.
See: https://github.com/apache/pinot/pull/9910 and https://github.com/trinodb/trino/blob/1866a23e3b0377144c1820de892c0de2762351a8/core/trino-main/src/main/java/io/trino/operator/aggregation/state/CovarianceState.java
Contributor guide
Research direction
Start by locating Pinot's current covar implementation and its tests; the issue does not name the files or entry point. Review the Apache Pinot pull request 9910 and Trino's CovarianceState.java for the numerical-stability approach, then verify that covariance results remain correct when the covariance is small.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100