[FEA] Avoid materializing temporary table in ORC chunked reader
Open
cuIO
feature request
libcudf
no-oom
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
In ORC chunked reader, the decoded stripes are materialized into a cudf table before it is splitt into multiple output chunks through slicing. Materializing such table is memory consuming. We can optimize memory usage in the chunked reader by avoiding that step altogether. By doing so, only part of the decoded stripes is materialized which is enough for one output chunk.
This requires some amount of work to rewrite `cudf::io::detail::column_buffer`.
Contributor guide
Assessment
This issue has not been assessed yet.