NVIDIA / NVIDIA/cudf

[FEA] Handle size overflow in nested columns by ORC chunked reader

Open
#15,848 0 comments 0 reactions 1 assignee Claimed by @ttnghia View on GitHub
cuIO feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In the current chunked ORC reader, only row counts of the top-level columns are considered when splitting the loaded stripes into subsets of stripes to avoid decoding more than 2B rows, which is cudf column size limit. The nested children columns are not taken into account.

We should consider row count at all nested levels.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.