NVIDIA / NVIDIA/cudf

[FEA] Spark-compatible iterative seeding for multi-column MurmurHash3

Open
#21,720 8 comments 0 reactions 1 assignee Claimed by @PointKernel View on GitHub
feature request libcudf Spark Velox
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

Velox's cuDF integration uses `cudf::hashing::murmurhash3_x86_32` to GPU-accelerate Spark's `hash_with_seed` function. Single-column hashing works correctly, but multi-column hashing produces different results than Spark due to how columns are combined.

***Spark's Murmur3Hash*** (iterative seeding):
```
h0 = murmur3(col0, seed)
h1 = murmur3(col1, h0) // previous hash becomes the seed
h2 = murmur3(col2, h1)
...
```

***cuDF's `murmurhash3_x86_32`*** (same-seed + hash_combine):
```
h0 = murmur3(col0, seed)
h1 = murmur3(col1, seed) // same original seed for every column
h2 = murmur3(col2, seed)
result = hash_combine(hash_combine(h0, h1), h2)
```
Each column is hashed with the same original scalar seed and then combined while Spark instead uses each column's hash result as the seed for the next column.

**Describe the solution you'd like**
`row_hasher::device_hasher` already accepts a `DeviceRowHasher` template parameter defaulting to `device_row_hasher`.
A new `device_row_hasher_iterative` class can slot in without modifying existing infrastructure — it just needs to implement the same `operator()(size_type row_index)` interface but with iterative seeding logic instead of `hash_combine`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.