NVIDIA / NVIDIA/cudf

[BUG] cuFile driver closing causes segfault upon program termination

Open
#17,121 4 comments 0 reactions 1 assignee Claimed by @kingcrimsontianyu View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

cuDF accesses the cuFile API via the [cufile_shim](https://github.com/rapidsai/cudf/blob/b8917229f8a2446c7e5f697475f76743a05e6856/cpp/src/io/utilities/file_io_utilities.cpp#L92) object that has a static storage duration, meaning its destructor is called after the main function returns. The `cuFileDriverClose()` internally calls CUDA API, resulting in UB (usually manifested as segfault), and therefore should not be called here. However, even in the absence of `cuFileDriverClose()`, cuFile will implicitly close the driver, during which process some CUDA calls are still made, likely causing segfault. The best way to clean up the resources needs to be revisited in the future.

For the time being, segfault cannot be completely avoided under the GDS I/O path, but at least `cuFileDriverClose()` should be removed from the destructor of cufile_shim.

Related issues from KvikIO:
https://github.com/rapidsai/kvikio/issues/497

**Steps/Code to reproduce bug**
Run any program using GDS I/O.

**Expected behavior**
Free of segmentation fault.

**Environment overview (please complete the following information)**
N/A

**Environment details**
N/A

**Additional context**
N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.