NVIDIA / NVIDIA/cudf

[BUG] Intermittent bug decompressing files in the parquet reader in 24.08.

Open
#16,661 4 comments 0 reactions 0 assignees View on GitHub
bug cuIO Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

This is a complex bug. It was initially seen running NDS benchmarks on H100 machines. We could not get it to reproduce on other machines, although later investigation leads up to believe this probably isn't actually H100 specific, but is more of a timing/race sort of issue that H100 perturbs into happening.

- During one of the NDS queries, we can get a fairly reproducible failure to decompress a parquet file, which sometimes manifests as an overt exception thrown by the reader "Error during decompression", but also seems to manifest as simply corrupt data, leading the benchmark to fail validation against the CPU.

- This reproduces in 24.08, but not 24.06. Also, it does not seem to reproduce from a clean local cudf build. Only from a `libcudf.so` that comes out of the `spark-rapids-jni` environment.

- We managed to get a reproducible case with a handful of files that are coalesced by Spark and handed to cudf. See attached.

- Running a process that has >= 2 threads that spin forever and randomly load from the selection of files, on an H100 machine will produce the exception reasonably quickly. Typically within < 1000 loads per thread (which only takes seconds - these files are tiny).

- These files are gzip compressed, which is somewhat out of the ordinary. After digging into this quite a bit, the hand-rolled `gpuinflate` implementation in cudf seemed to be a likely candidate. It does some odd things passing messages between warps that seem like they're probably UB. And `compute-sanitizer` does complain about races in the kernel. I wasn't able to narrow it all the way down though.

- Instead, I tried swapping `gpuinflate` out entirely and just using nvcomp for gzip. With one caveat, this makes the problem go away. The machine that repros the issue remains fine after hundreds of thousands of loads across multiple threads.

- The big caveat from above is that the nvcomp gzip decompressor seems to have explicit trouble with the file `dbgdump586530430.parquet`. It quietly fails to decode the first data page, but doesn't produce an error. It seems likely that this is a different issue entirely and I've sent a repro to nvcomp. If I remove this file from the rotation and run the test, there are no issues.

So, while not 100% conclusive, it seems like the gpuinflate() implementation in cudf has some race/timing/synchronization issues which occasionally can cause failures, with H100 being a good way to do it.

I put up a branch with my (very much quick-and-dirty) nvcomp integration.
https://github.com/nvdbaranec/cudf/tree/nvcomp_2408_integration

For some reason, github isn't letting me upload the archive (it's only 97k). Ping me on slack if you'd like to see it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.