NVIDIA / NVIDIA/cudf

[FEA] Implement column-wise hashes

Open
#14,484 3 comments 0 reactions 0 assignees View on GitHub
feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

cudf columns are mutable and therefore do not (or should not) implement `__hash__` (in the same way that numpy arrays do not do so).

_However_, there are circumstances under which we would nonetheless like to be able to compute a hash of a column:

1. When the wrapping object is actually an immutable one (for example `Index` objects) and so `__hash__` is safe;
2. When tokenizing keys for dask task graphs (see https://github.com/rapidsai/cudf/pull/13695), where the objects may be mutable, but the required semantics are "two objects that compare equal should hash the same". This enables dask to perform some amount of optimisation on the task graph for repeated execution and task merging.

**Describe the solution you'd like**

I would like to be able to hash a column with a libcudf call and receive a single $k$-bit hash. The first point above does not need to worry excessively about collisions, and python hash values are 64bit ints, so a 64-bit murmur- or xx-hash is likely sufficient. For dask, collisions are more problematic, so a 128-bit md5 would be better (this is what dask uses for pandas dataframes).

**Describe alternatives you've considered**

Compute row-wise hashes of columns (on dataframes) to produce a single column of hashes and then copy to host to hash there.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.