[BUG] NDS-H data generator stalls after scale factor 150
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Currently, `libcudf` supports upto 32-bit integers. Hence, the NDS-H data generator which is built using `libcudf` can only generate upto 2,147,483,647 rows before it hits the row limit. For NDS-H, this translates to a scale factor of ~350 (The `lineitem` table, which is the largest NDS-H table, has ~6M rows, so the largest `lineitem` table that can be generated is of scale factor 350). But, when we tried running the NDS-H data generator with scale factor > 150, it stalled indefinitely. The highest scale factor that it was able to successfully generate was 150. This was observed both with un-managed and managed memory, and the execution stalled even though there was enough memory left.
**Steps/Code to reproduce bug**
Run the data generator with scale factor > 150
**Expected behavior**
The NDS-H data generator should be able to generate upto scale factor ~350.
**Environment overview (please complete the following information)**
- Environment location: NVIDIA H100 and NVIDIA GH200
- Method of cuDF install: Built from source, branch-24.10
Contributor guide
Assessment
This issue has not been assessed yet.