[FEA] Parquet reader code cleanup, re: nested columns vs columns with lists.
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
In the parquet reader there are two similar-sounding but distinct pieces of terminology:
- Nested columns. This is the same as in the cudf sense. Anything involving structs or lists at any level.
- Nested hierarchies. This only involves columns (or _parts_ of columns) that contain lists (represented via repetition levels).
This causes confusion and bugs for a couple of reasons. A given (cudf) output column can contain both nested and non-nested hierarchies. For example:
```
A (struct)
/ \
B C (list)
|
D (int)
```
This single output column contains two separate input column hierarchies. A->B and A->C->D. A->B does not contain repetition data and therefore is not a nested hierarchy. A->C->D does contain repetition data and does constitute a nested hierarchy. However they are _both_ nested in the cudf sense (more than 1 level deep).
We handle these two fundamental situations differently during the decoding process. So if the two concepts get confused it can easily cause bugs.
It would be great to do a pass that cleans this up in a comprehensive way.
Contributor guide
Assessment
This issue has not been assessed yet.