apache / apache/gluten

[GLUTEN][VL] Expose Binary Sketch(hll) for Persistent Storage and Merging

Open
#9,202 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Gluten currently implements the `approx_count_distinct` function, leveraging HyperLogLog (HLL) or similar approximate cardinality estimation algorithms. However, the underlying binary sketch representation generated by these algorithms is not exposed to the user.

**Problem:**

This limitation prevents users from:
1. **Persistently storing the sketches:** The inability to serialize and store the binary sketch hinders offline analysis and long-term data aggregation.
2. **Merging sketches:** Merging sketches from different datasets or partitions is essential for accurate cardinality estimation across larger datasets. Without access to the binary representation, this operation is not feasible.
3. **Performing custom analysis:** Users requiring advanced cardinality analysis or integration with external systems are restricted by the lack of direct access to the sketch.

**Proposed Solution:**

Expose the binary sketch representation as a `BINARY` or `BYTE_ARRAY` type. This would allow users to:
_1. Retrieve the binary sketch
2. Store the binary sketch
3. Merge binary sketches
4. Estimate cardinality from the merged sketch_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.