crate / crate/crate

Improve performance in HyperLogLogDistinctAggregation

Open
#16,439 0 comments 0 reactions 0 assignees View on GitHub
feature: performance feature: sql: aggregations
Dominant language
Java
Stars
4.4k
Forks
616
Avg merge
14h 49m
Merged PRs (30d)
206

Description

HyperLogLogDistinctAggregation contains the following code executed for each document that matches the top-level query:
```
BytesRef ref = super.values.lookupOrd(super.values.nextOrd());
byte[] bytes = NetworkUtils.formatIPBytes(ref).getBytes(StandardCharsets.UTF_8);

var hash = state.isAllOn4_1() ?
MurmurHash3.hash64(bytes, 0, bytes.length)
: MurmurHash3.hash128(bytes, 0, bytes.length, 0, super.hash128).h1;
state.addHash(hash);
```

For low-cardinality fields it will be more efficient to calculate the hashes of all values in the terms dictionary up-front, store these in an array, and use the per-document ordinal values to look up the pre-calculated hash, rather than doing a `lookupOrd` and re-calculating the hash on every doc.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.