apache / apache/datasketches-java

can we use stable sort for tdigest

Open
#739 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
958
Forks
226
Avg merge
3d 9h
Merged PRs (30d)
8

Description

the sort on centroids (v,w) -> (value, weight) in tdigest merge uses the sort based on value (v) and index -> therefore if we have centroids like (200,1)(100,3)(300,2)(100,1) then at merge it can be sorted to 2 states :

1. (100,1)(100,3)(200,1)(300,2)
2. (100,3)(100,1)(200,1)(300,2)

due to which the merging result can differ

other than that we can use sorting based on value (v) and weight (w) as in
https://github.com/tdunning/t-digest/blob/main/core/src/main/java/com/tdunning/math/stats/Sort.java

and this will make the sort stable and remove variance in result

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the t-digest merge logic where centroids are sorted by value and index. Compare that ordering with core/src/main/java/com/tdunning/math/stats/Sort.java referenced in the issue, then verify that equal-value centroids are ordered deterministically and produce consistent merge results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.