[CH]fix precision issue in statistical aggregation functions
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
CH (ClickHouse)
### Bug description
The precision is different between Spark and Gluten+CH back end for the following statistical aggregation functions:
var_samp
var_pop
covar_samp
covar_pop
stddev_samp
stddev_pop
corr
variance
So, I want to introduce following changes:
1. use varSampStable,varPopStable ... to replace varSamp, varPop ...
2. use fixed nullable property for corr like aggregation's input
3. convert Infinite to Null to keep compatiable with Spark
### Spark version
Spark-3.3.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start by locating the ClickHouse implementations of var_samp, var_pop, covar_samp, covar_pop, stddev_samp, stddev_pop, corr, and variance, then compare their behavior with Spark 3.3.x. Done means stable aggregation functions are used, corr preserves the input's nullable property, and infinite results become null for Spark compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, spark
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100