NVIDIA / NVIDIA/cudf

[BUG] Dictionary encoding silently corrupts string columns in Parquet writer at row group boundaries

Open
#22,299 0 comments 0 reactions 0 assignees View on GitHub
bug cuIO strings
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

cuDF version: 26.02.01

Description:
Writing a 400M row DataFrame with string columns produces silent data corruption.
String values are silently replaced with empty strings. No error or warning is raised.
Numeric columns are not affected.

Setting use_dictionary=False eliminates the corruption entirely, confirming the bug is in dictionary encoding across
row group boundaries.

Appears that more row groups is more likely to cause corruption. Setting row_group_size_rows below 20M results in corrupt df when reading back in, but not if above.

DataFrame: 400M rows, 9 columns (5 string, 4 numeric)
Total string data: ~15 GB across 5 columns
Unique values per string column: ~300

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.