facultyai / facultyai/lens

Consider removal of t-digest computation

Aberta
#10 0 comentários 0 reações 0 responsáveis Ver no GitHub
discussion enhancement good first issue
Linguagem predominante
Python
Estrelas
99
Forks
8
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

Right now the [t-digest](https://github.com/tdunning/t-digest) computation (done using a [python t-digest implementation](https://github.com/CamDavidsonPilon/tdigest)) takes most of the time in generating a summary. The initial motivation to include it was for it to contain an approximation of the histogram information, but we are also computing a fixed-bin-width histogram so it is of limited value. The t-digest information is used in the explorer for:

- arbitrary bin width histograms.
- building percentile functions in `lens.Summary` that get used to plot a CDF in `lens.Explorer`.

We have to consider whether these two features are important enough and whether we can use other approaches to substitute this information.

Having an adaptively binned histogram (through, e.g., bayesian blocks) would go a long way to replacing the t-digest for our exploration needs.

A significant advantage of a t-digest is that it can be updated in a chunked manner, but we are not currently using that.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.