apache / apache/pinot

Refactor covariance function to numerical stable version

Open
#9,936 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

As @snleee has pointed out, the current implementation for covar is not numerical stable and may subject to https://en.wikipedia.org/wiki/Catastrophic_cancellation if the covariance value itself is small.
See: https://github.com/apache/pinot/pull/9910 and https://github.com/trinodb/trino/blob/1866a23e3b0377144c1820de892c0de2762351a8/core/trino-main/src/main/java/io/trino/operator/aggregation/state/CovarianceState.java

Contributor guide

Open the contributing guide

Research direction

Start by locating Pinot's current covar implementation and its tests; the issue does not name the files or entry point. Review the Apache Pinot pull request 9910 and Trino's CovarianceState.java for the numerical-stability approach, then verify that covariance results remain correct when the covariance is small.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.