NVIDIA / NVIDIA/cudf

[BUG] Degraded to_parquet performance on lustre

Open
#23,751 1 comment 0 reactions 0 assignees View on GitHub
? - Needs Triage bug
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
When bumping from Rapids 25.10 to 26.08 (it is a big jump). We see degraded performance from `to_parquet`. when reading/writing from many GPUs to a lustre based distributed filesystem.

In one case the avg time to write a dataframe went from ~0.3s to ~1.5s and the standard deviation went from 0.02s to 7.5s.
On end to end deduplication workflows this has a 20-40% impact on overall runtime.
Side note: The issue isn't as apparent when testing on a single GPU, I think the combination of multiple GPUs all hitting the filesystem at the same time with these new default options is what makes perf worse.

**Steps/Code to reproduce bug**
1. On a machine with 8 or more GPUs, setup 1 process per GPU each writing a dataframe to disk (lustre).
2. The specific schema I used was `int64, list[260]`

**Expected behavior**
Based on the pattern it seems related to the interaction of many threads with the lustre filesystem and I explores disabling the changes in https://github.com/rapidsai/kvikio/pull/863.

Setting `KVIKIO_AUTO_DIRECT_IO_WRITE=0` helps performance go back to 25.10 levels.

So either the auto direct write feature doesn't play well with lustre or it's interaction with #23231 severely degrades performance on lustre like filesystems where multiple threads per GPU and multiple GPUs are simultaneously writing to the filesystem.

**Environment overview (please complete the following information)**
- Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
- Method of cuDF install: [conda, Docker, or from source]
- If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the eight-or-more-GPU workload described in the issue with the int64 and list[260] schema on Lustre, comparing default behavior with KVIKIO_AUTO_DIRECT_IO_WRITE=0. Read kvikio PR #863 and cuDF issue #23231 for the implicated changes; done means isolating the regression and verifying that the affected write path returns to the 25.10 performance level without breaking other filesystems.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.