[FEA] Implement parallel reading of Parquet footers without using the additional thread pool
Open
cuIO
feature request
libcudf
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
https://github.com/rapidsai/cudf/pull/17957 optimizes ingest of multiple Parquet files, using an internal thread pool to issue read requests. Parallelism of the read operations is already implemented in kvikIO, so we should not need to launch these reads from multiple threads.
With an [async read interface](https://github.com/rapidsai/cudf/pull/18018), we could read all footers in a loop from a single thread.
Contributor guide
Assessment
This issue has not been assessed yet.