NVIDIA / NVIDIA/cudf

[FEA] Avoid materializing temporary table in ORC chunked reader

Open
#15,769 0 comments 0 reactions 1 assignee Claimed by @ttnghia View on GitHub
cuIO feature request libcudf no-oom
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In ORC chunked reader, the decoded stripes are materialized into a cudf table before it is splitt into multiple output chunks through slicing. Materializing such table is memory consuming. We can optimize memory usage in the chunked reader by avoiding that step altogether. By doing so, only part of the decoded stripes is materialized which is enough for one output chunk.

This requires some amount of work to rewrite `cudf::io::detail::column_buffer`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.