NVIDIA / NVIDIA/cudf

[FEA] Hash values in nested columns

Open
#5,640 3 comments 0 reactions 0 assignees View on GitHub
feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
2d 22h
Merged PRs (30d)
274

Description

Hashing categorical features to a fixed number of bins is a common preprocessing operation, particularly for tabular deep learning models where memory requirements scale with the number of bins. For extension of CuDF to nested columns, it would be helpful if calls to `Series.hash_values` hashed the values *in* each element's list and not the list itself. This would allow categorical hashing to extend to multi-hot categorical features.

As an example:
```
df = cudf.DataFrame({'a': [[0, 1, 2], [3, 4], [], [5], [6, 7, 8, 9]]})
df['a'].hash_values()

# not sure what this representation will
# print like but some like this
[[ 29140149, -247539971, 1683430573], [1098043756, 1851360991], [],
[100260016], [154726282, -1778135556, -1793932552, 246633392]]

df['a'].hash_values() % 4
[[1, 1, 1], [0, 3], [], [0], [2, 0, 0, 0]]
```

**Additional context**
Necessary for extension of [NVTabular HashBucket op](https://github.com/NVIDIA/NVTabular/blob/9f8216a89d565e00d8356ffef62f4437f3e2dee3/nvtabular/ops.py#L498) to multi-hot categorical data

Contributor guide

Open the contributing guide

Research direction

Start by tracing the cuDF Series.hash_values entry point and how it handles nested list columns. Use the issue's multi-hot example and NVTabular HashBucket context to define expected per-element results, including empty lists; done means nested values are hashed individually and modulo binning works as shown.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.