NVIDIA / NVIDIA/cudf

[BUG] NDS-H data generator stalls after scale factor 150

Open
#16,837 0 comments 0 reactions 0 assignees View on GitHub
bug libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Currently, `libcudf` supports upto 32-bit integers. Hence, the NDS-H data generator which is built using `libcudf` can only generate upto 2,147,483,647 rows before it hits the row limit. For NDS-H, this translates to a scale factor of ~350 (The `lineitem` table, which is the largest NDS-H table, has ~6M rows, so the largest `lineitem` table that can be generated is of scale factor 350). But, when we tried running the NDS-H data generator with scale factor > 150, it stalled indefinitely. The highest scale factor that it was able to successfully generate was 150. This was observed both with un-managed and managed memory, and the execution stalled even though there was enough memory left.

**Steps/Code to reproduce bug**
Run the data generator with scale factor > 150

**Expected behavior**
The NDS-H data generator should be able to generate upto scale factor ~350.

**Environment overview (please complete the following information)**
- Environment location: NVIDIA H100 and NVIDIA GH200
- Method of cuDF install: Built from source, branch-24.10

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.