NVIDIA / NVIDIA/cudf

[QST] [libcudf & nsys] Unstable Runtime during Parquet Reading: due to I/O Futex and munmap

Open
#18,353 3 comments 0 reactions 0 assignees View on GitHub
cuIO libcudf question
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

I have repeatedly observed unstable runtime when reading Parquet files, where the reading time occasionally exceeds the average. To investigate this issue, I conducted an nsys profiling analysis. The setup is the same as my previous posts as #18278

It's possible that this instability is a profiling-related problem. However, I am unable to confirm this at present.

During the nsys profiling, I frequently encountered two time-consuming issues: I/O futex and Munmap. I am primarily interested in understanding how to eliminate these issues users require a stable performance evaluation without these "noisy" factors. Additionally, I would like to know if you have also encountered these problems. If you have suggestions for a better setup to avoid these time-consuming issues, your input would be greatly appreciated.

# I/O Futex

On my end, I/O futex issues are quite common. For instance, a read operation involving futex from KVIKIO often takes significantly longer than other normal read operations. It appears that this operation is synchronizing with a write call. This typically occurs when reading compressed data, prior to the execution of decoding and decompression kernels.

![Image](https://github.com/user-attachments/assets/52a2f035-8564-4c00-bbf0-02feda447006)

![Image](https://github.com/user-attachments/assets/1c857d04-0b57-4f2c-88ac-00ada6d33bf8)

# Munmap

Munmap issues are less frequent. They occur during the resource deallocation process at the very end when objects are being deconstructed `~memory_mapped_source()`. It is not entirely clear to me whether this unmapping operation is synchronizing with other processes.

![Image](https://github.com/user-attachments/assets/077b7287-e60e-4744-a0cf-fe945f12ede1)

---

Finally, if these issues do not impact performance, are merely nsys overheads, or are the result of a misconfiguration in my setup, I will close this inquiry. Thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.