NVIDIA / NVIDIA/cudf

Shared-memory groupby hangs when multi-column packing exceeds the single-column compatibility guard

Open
#23,829 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

The shared-memory hash groupby can hang forever when the host-side compatibility check passes but the device-side packing cannot fit all aggregation columns in one pass. This is a follow-on to #17853: the guard added by PR #17851 checks only the single-column data footprint, while the kernel needs room for every column's data plus its validity mask plus 16-byte alignment padding.

**Mechanism**

`is_shared_memory_compatible` (`cpp/src/groupby/hash/compute_single_pass_aggs.cu:26-36`) accepts the shared-memory path when `data_buffer_size >= size * GROUPBY_CARDINALITY_THRESHOLD` for each column individually. The kernel's packing step `calculate_columns_to_aggregate` (`cpp/src/groupby/hash/compute_shared_memory_aggs.cu:121-153`) instead needs, for all columns together, `sum_i(round_up16(width_i*C) + round_up16(C))` bytes where C is the block cardinality (up to 128). When that sum does not fit, the loop breaks without advancing and the driver `while (col_end < num_cols)` at line 302 has no progress condition, so it re-runs the identical break point forever; any single spinning block hangs the whole kernel.

For int64 aggregations at cardinality 128 one column needs `round16(8*128) + round16(128) = 1152` bytes, but the guard only requires `avail - offsets >= 1024`. Any configuration with per-column slack in `[1024, 1152)` passes the host check and then spins.

**Reproduced on RTX 5060 Ti (sm_120), libcudf 26.10 nightly**

One int64 key column with all-distinct keys plus k int64 value columns aggregated with SUM:

| rows | k (number of value columns) |
|---|---|
| 27648 | 868-870 hang, neighbors ok |
| 23040 | 1082-1086 hang, neighbors ok |
| 18432 | 1402-1406 hang, neighbors ok |
| 13824 | 1934 hangs |

Each hanging config deterministically times out (60 s); configs just outside the narrow windows return normally, matching the computed window boundary exactly. A cuda-gdb backtrace of a live hang shows the main thread blocked in `cudaFree` inside `compute_single_pass_aggs`, i.e. the launched `single_pass_shmem_aggs_kernel` never retires.

The same window arithmetic predicts hangs on H100/B200 at different occupancy points (for example 6 int64 aggregations at blocksPerSM=16 on 228KB parts).

**Suggested fix**

Tighten the host check to the true worst-case total the packer needs at cardinality 128: require `sum_i(round_up16(width_i*128) + round_up16(128)) <= data_buffer_size` (falling back to the existing global-memory path otherwise). That is conservative for blocks whose cardinality is below 128 but eliminates the spin.

Contributor guide

Open the contributing guide

Research direction

Read is_shared_memory_compatible in cpp/src/groupby/hash/compute_single_pass_aggs.cu:26-36 and calculate_columns_to_aggregate in cpp/src/groupby/hash/compute_shared_memory_aggs.cu:121-153, then inspect the driver loop at line 302. Verify the host check accounts for every column's rounded data and validity-mask space at cardinality 128. Done means configurations that previously hung fall back to the global-memory path without the kernel spinning.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.