[FEA] Spark-compatible iterative seeding for multi-column MurmurHash3
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Velox's cuDF integration uses `cudf::hashing::murmurhash3_x86_32` to GPU-accelerate Spark's `hash_with_seed` function. Single-column hashing works correctly, but multi-column hashing produces different results than Spark due to how columns are combined.
***Spark's Murmur3Hash*** (iterative seeding):
```
h0 = murmur3(col0, seed)
h1 = murmur3(col1, h0) // previous hash becomes the seed
h2 = murmur3(col2, h1)
...
```
***cuDF's `murmurhash3_x86_32`*** (same-seed + hash_combine):
```
h0 = murmur3(col0, seed)
h1 = murmur3(col1, seed) // same original seed for every column
h2 = murmur3(col2, seed)
result = hash_combine(hash_combine(h0, h1), h2)
```
Each column is hashed with the same original scalar seed and then combined while Spark instead uses each column's hash result as the seed for the next column.
**Describe the solution you'd like**
`row_hasher::device_hasher` already accepts a `DeviceRowHasher` template parameter defaulting to `device_row_hasher`.
A new `device_row_hasher_iterative` class can slot in without modifying existing infrastructure — it just needs to implement the same `operator()(size_type row_index)` interface but with iterative seeding logic instead of `hash_combine`.
Contributor guide
Assessment
This issue has not been assessed yet.