Shared-memory groupby hangs when multi-column packing exceeds the single-column compatibility guard
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
The shared-memory hash groupby can hang forever when the host-side compatibility check passes but the device-side packing cannot fit all aggregation columns in one pass. This is a follow-on to #17853: the guard added by PR #17851 checks only the single-column data footprint, while the kernel needs room for every column's data plus its validity mask plus 16-byte alignment padding.
**Mechanism**
`is_shared_memory_compatible` (`cpp/src/groupby/hash/compute_single_pass_aggs.cu:26-36`) accepts the shared-memory path when `data_buffer_size >= size * GROUPBY_CARDINALITY_THRESHOLD` for each column individually. The kernel's packing step `calculate_columns_to_aggregate` (`cpp/src/groupby/hash/compute_shared_memory_aggs.cu:121-153`) instead needs, for all columns together, `sum_i(round_up16(width_i*C) + round_up16(C))` bytes where C is the block cardinality (up to 128). When that sum does not fit, the loop breaks without advancing and the driver `while (col_end < num_cols)` at line 302 has no progress condition, so it re-runs the identical break point forever; any single spinning block hangs the whole kernel.
For int64 aggregations at cardinality 128 one column needs `round16(8*128) + round16(128) = 1152` bytes, but the guard only requires `avail - offsets >= 1024`. Any configuration with per-column slack in `[1024, 1152)` passes the host check and then spins.
**Reproduced on RTX 5060 Ti (sm_120), libcudf 26.10 nightly**
One int64 key column with all-distinct keys plus k int64 value columns aggregated with SUM:
| rows | k (number of value columns) |
|---|---|
| 27648 | 868-870 hang, neighbors ok |
| 23040 | 1082-1086 hang, neighbors ok |
| 18432 | 1402-1406 hang, neighbors ok |
| 13824 | 1934 hangs |
Each hanging config deterministically times out (60 s); configs just outside the narrow windows return normally, matching the computed window boundary exactly. A cuda-gdb backtrace of a live hang shows the main thread blocked in `cudaFree` inside `compute_single_pass_aggs`, i.e. the launched `single_pass_shmem_aggs_kernel` never retires.
The same window arithmetic predicts hangs on H100/B200 at different occupancy points (for example 6 int64 aggregations at blocksPerSM=16 on 228KB parts).
**Suggested fix**
Tighten the host check to the true worst-case total the packer needs at cardinality 128: require `sum_i(round_up16(width_i*128) + round_up16(128)) <= data_buffer_size` (falling back to the existing global-memory path otherwise). That is conservative for blocks whose cardinality is below 128 but eliminates the spin.
Contributor guide
Research direction
Read is_shared_memory_compatible in cpp/src/groupby/hash/compute_single_pass_aggs.cu:26-36 and calculate_columns_to_aggregate in cpp/src/groupby/hash/compute_shared_memory_aggs.cu:121-153, then inspect the driver loop at line 302. Verify the host check accounts for every column's rounded data and validity-mask space at cardinality 128. Done means configurations that previously hung fall back to the global-memory path without the kernel spinning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100