ClickHouse / ClickHouse/ClickHouse

rankCorr returns incorrect coefficient with large number of tied/duplicate values

Open
#113,050 3 comments 0 reactions 0 assignees View on GitHub
comp-aggregate-functions external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

Project: Data Statistical Verification Scripts
Description: Feature correlation statistical computing.

### Describe what's wrong

The aggregate function `rankCorr` claims to calculate standard Spearman's rank correlation coefficient, but it produces severely biased, even sign-reversed results when input columns contain a large amount of duplicate (tied) values.

Root observation from data comparison:
1. The internal function `computeRanksAndTieCorrection` correctly calculates mid-ranks and tie correction penalty terms for duplicates.
2. However, the final calculation discards all tie correction values and only uses the simplified formula:
$$\rho = 1 - \frac{6\sum d^2}{n(n^2-1)}$$
This simplified formula is only mathematically valid when there are zero tied values in both columns. Massive duplicates break its core mathematical assumption, leading to wrong magnitude and inverted correlation sign.

### Does it reproduce on the most recent release?

Yes

### How to reproduce

## Version information
Tested versions: 26.7.1.1315 stable releases

## Test data background
I have a dataset with two columns `f` and `r` as attached:
- Column `f`: ~72% rows are exactly 0 (heavy ties), other discrete values: ±0.9999, 0.99995, small fractional numbers
- Column `r`: continuous tiny floating noise values

[result.csv](https://github.com/user-attachments/files/30643055/result.csv)

## Comparison results
1. Standard calculation (Python scipy.stats.spearmanr, industry standard): **-0.0506**
2. ClickHouse native `rankCorr(f, r)` on identical dataset: **0.1520**
- Error: sign flipped, absolute value differs drastically.

### Expected behavior

rankCorr should follow the standard Spearman definition consistent with SciPy / R / Pandas:
Calculate mid-rank for all tied records
Compute Pearson correlation based on two groups of mid-rank values
This method can fully eliminate calculation bias caused by duplicate values.

### Error message and/or stacktrace

_No response_

### Related issues and pull requests

_No response_

### Additional context

1. Code location reference
File path: src/AggregateFunctions/AggregateFunctionRankCorrelation.cpp
The program calls computeRanksAndTieCorrection, but uses std::ignore to abandon the precomputed tie correction parameters, then only runs the simplified formula without any tie compensation.
2. Statistical industry consensus
This simplified formula is widely known to fail on tied data; many data engineering articles mention this common pitfall of Spearman calculation.
3. Documentation defect
Official ClickHouse docs do not remind users of this critical limitation of rankCorr.
4. Temporary workaround
I have found a correct but low-efficiency solution: calculate mid-rank through window functions manually, then use corr() to get standard Spearman value. The SQL example is as follows:
sql
WITH ranked_data AS (
SELECT
f, r,
rank() OVER (ORDER BY f) + (count() OVER (PARTITION BY f) - 1) / 2.0 AS mid_rank_f,
rank() OVER (ORDER BY r) + (count() OVER (PARTITION BY r) - 1) / 2.0 AS mid_rank_r
FROM test_rankcorr_ties
)
SELECT corr(mid_rank_f, r) AS standard_spearman FROM ranked_data;
This method produces results consistent with scipy, but performs much worse than the native aggregate rankCorr on large tables.

Contributor guide

Open the contributing guide

Research direction

Start with src/AggregateFunctions/AggregateFunctionRankCorrelation.cpp, especially computeRanksAndTieCorrection and the call site that discards its tie correction values. Reproduce the attached-data result and compare rankCorr with SciPy or the documented mid-rank Pearson definition. Done means tied inputs produce the standard Spearman coefficient, including the expected sign and magnitude.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, cpp, sql
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.