NVIDIA / NVIDIA/cudf

[FEA] Parquet reader code cleanup, re: nested columns vs columns with lists.

Open
#11,793 0 comments 0 reactions 0 assignees View on GitHub
0 - Backlog cuIO libcudf proposal
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In the parquet reader there are two similar-sounding but distinct pieces of terminology:

- Nested columns. This is the same as in the cudf sense. Anything involving structs or lists at any level.
- Nested hierarchies. This only involves columns (or _parts_ of columns) that contain lists (represented via repetition levels).

This causes confusion and bugs for a couple of reasons. A given (cudf) output column can contain both nested and non-nested hierarchies. For example:
```
A (struct)
/ \
B C (list)
|
D (int)
```
This single output column contains two separate input column hierarchies. A->B and A->C->D. A->B does not contain repetition data and therefore is not a nested hierarchy. A->C->D does contain repetition data and does constitute a nested hierarchy. However they are _both_ nested in the cudf sense (more than 1 level deep).

We handle these two fundamental situations differently during the decoding process. So if the two concepts get confused it can easily cause bugs.

It would be great to do a pass that cleans this up in a comprehensive way.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.