[FEA] Implement column-wise hashes
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
cudf columns are mutable and therefore do not (or should not) implement `__hash__` (in the same way that numpy arrays do not do so).
_However_, there are circumstances under which we would nonetheless like to be able to compute a hash of a column:
1. When the wrapping object is actually an immutable one (for example `Index` objects) and so `__hash__` is safe;
2. When tokenizing keys for dask task graphs (see https://github.com/rapidsai/cudf/pull/13695), where the objects may be mutable, but the required semantics are "two objects that compare equal should hash the same". This enables dask to perform some amount of optimisation on the task graph for repeated execution and task merging.
**Describe the solution you'd like**
I would like to be able to hash a column with a libcudf call and receive a single $k$-bit hash. The first point above does not need to worry excessively about collisions, and python hash values are 64bit ints, so a 64-bit murmur- or xx-hash is likely sufficient. For dask, collisions are more problematic, so a 128-bit md5 would be better (this is what dask uses for pandas dataframes).
**Describe alternatives you've considered**
Compute row-wise hashes of columns (on dataframes) to produce a single column of hashes and then copy to host to hash there.
Contributor guide
Assessment
This issue has not been assessed yet.