[GLUTEN][VL] Expose Binary Sketch(hll) for Persistent Storage and Merging
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Gluten currently implements the `approx_count_distinct` function, leveraging HyperLogLog (HLL) or similar approximate cardinality estimation algorithms. However, the underlying binary sketch representation generated by these algorithms is not exposed to the user.
**Problem:**
This limitation prevents users from:
1. **Persistently storing the sketches:** The inability to serialize and store the binary sketch hinders offline analysis and long-term data aggregation.
2. **Merging sketches:** Merging sketches from different datasets or partitions is essential for accurate cardinality estimation across larger datasets. Without access to the binary representation, this operation is not feasible.
3. **Performing custom analysis:** Users requiring advanced cardinality analysis or integration with external systems are restricted by the lack of direct access to the sketch.
**Proposed Solution:**
Expose the binary sketch representation as a `BINARY` or `BYTE_ARRAY` type. This would allow users to:
_1. Retrieve the binary sketch
2. Store the binary sketch
3. Merge binary sketches
4. Estimate cardinality from the merged sketch_
Contributor guide
Assessment
This issue has not been assessed yet.