[FEA] Expose per-row partition numbers without materializing columns
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
`cudf::hash_partition` computes a partition assignment and materializes every input column in partition order in one call. Query engines can compute the assignment once from join keys, then materialize and migrate join keys and start executing the join before materializing and migrating payload columns.
Issue [#20882](https://github.com/rapidsai/cudf/issues/20882) and PR [#20891](https://github.com/rapidsai/cudf/pull/20891) separate partition computation from table materialization, but their proposed output is a set of per-partition gather maps.
PR #20891 first computes the per-row partition numbers, then performs additional scan and scatter work to convert them into output-to-input gather indices. Materializing columns from those indices reads the original rows in partition order, causing non-coalesced global-memory reads
**Describe the solution you'd like**
Add an API that returns one partition-number map:
```text
partition_map[input_row] = partition_id
```
**Describe alternatives you've considered**
The per-partition gather maps proposed by #20882 can materialize each partition independently, but GQE would need additional work to recover the original per-row partition assignment.
gather maps can be constructed from the partition-number map using existing cuDF operations. A caller can create a sequence of original row indices, partition that sequence with `cudf::partition` using the partition-number map, and slice the resulting index column at the returned partition offsets. This makes gather-map construction an
optional cost for callers that need it. Starting from only the per-partition gather maps, recovering the original per-row partition assignment requires an inverse scatter.
**Additional context**
Feature would allow for better compute communicator overlap for multi-GPU execution.
Contributor guide
Research direction
Start by reading cudf::hash_partition and cudf::partition, then review issue #20882 and PR #20891 to understand the existing partition and gather-map approaches. Done means a public API returns partition_map[input_row] = partition_id without materializing input columns, while preserving the stated multi-GPU use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100