apache / apache/gluten

[CH]fix precision issue in statistical aggregation functions

Open
#5,215 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

CH (ClickHouse)

### Bug description

The precision is different between Spark and Gluten+CH back end for the following statistical aggregation functions:
var_samp
var_pop
covar_samp
covar_pop
stddev_samp
stddev_pop
corr
variance

So, I want to introduce following changes:
1. use varSampStable,varPopStable ... to replace varSamp, varPop ...
2. use fixed nullable property for corr like aggregation's input
3. convert Infinite to Null to keep compatiable with Spark

### Spark version

Spark-3.3.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the ClickHouse implementations of var_samp, var_pop, covar_samp, covar_pop, stddev_samp, stddev_pop, corr, and variance, then compare their behavior with Spark 3.3.x. Done means stable aggregation functions are used, corr preserves the input's nullable property, and infinite results become null for Spark compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, spark
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.