[FEA] Handle size overflow in nested columns by ORC chunked reader
Open
cuIO
feature request
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
In the current chunked ORC reader, only row counts of the top-level columns are considered when splitting the loaded stripes into subsets of stripes to avoid decoding more than 2B rows, which is cudf column size limit. The nested children columns are not taken into account.
We should consider row count at all nested levels.
Contributor guide
Assessment
This issue has not been assessed yet.