[FEA] Automatically calculate appropiate number of hash partitions

Open
#88 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
data-engineering

Research direction

Start with examples/dask-nvtabular-criteo-benchmark.py, especially the hash-partition and host-memory settings around lines 45-54. Read how the benchmark currently chooses per-column values, then investigate the proposed parquet dictionary-encoding metadata approach. Done means customers no longer need to tune hash partitions and host-memory placement per column manually.

Written by the indexing model from the issue text.

Description

MultiGPU

Is your feature request related to a problem? Please describe.

The multigpu criteo benchmark is hardcoding the best number of hash partitions for each categorical variable:
https://github.com/NVIDIA/NVTabular/blob/2dd4cbc94e074d2a7a319dcf05ff249c7cdec3b3/examples/dask-nvtabular-criteo-benchmark.py#L45-L54
as well as specifying which columns should be stored in host memory

Describe the solution you'd like
We should automatically figure out the best number of hash partitions to use, and not require customers to know how to tune nvtabular on a per column basis

Additional context
One potential way of doing this for parquet files is to leverage the dictionary encoding metadata. We could also dynamically increase the number of hash partitions at runtime with some effort.

Dominant language
Python
Stars
1.2k
Forks
149
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-Merlin/NVTabular

All issues in NVIDIA-Merlin/NVTabular

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.