apache / apache/arrow

[C++][Python] Adding data to tdigest in pyarrow

Open
#35,508 27 comments 2 reactions 1 assignee Claimed by @gitmodimo View on GitHub
Component: C++ Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

Requests for the python implementation of the t-digest algorithm:

1. Currently it is only possible to compute one t-digest from a data array and return a given percentile. There doesn't seem to be an option to continuously 'add' data to the digest, as described in the clustering algorithm 2: https://arxiv.org/pdf/1902.04023.pdf. However I think this is happening under the hood, as reducing the buffer size, I assume, converts it the clustering algorithm variant. Would it be possible to include a a function such as pyarrow.compute.tdigest.add(), where you can add data points incrementally (as in the case of data streaming)?

2. Is it possible to alter the scale function used in the implementation? I'm not sure which scale function you have implemented but it would be nice to have some control over this!

Thanks

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.