NVIDIA / NVIDIA/cudf

Consider changing the `column_metadata` expectations when converting list types to arrow

Open
#16,600 0 comments 0 reactions 0 assignees View on GitHub
improvement libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Currently the Arrow conversion APIs use a `column_metadata` object to pass through information on what names to use for columns in the generated arrow table (no other metadata is currently present). Perhaps the most important use case of this is when struct types are present since struct fields are named in arrow while they are defined purely by their order in libcudf. The `column_metadata` struct is simply a name and a vector of `column_metadata` associated with the children of the column. The children list is empty for non-nested types, whereas for nested types like lists and structs there is one entry per child. For struct types, this 1-1 mapping is perfectly natural. For list types, however, it is somewhat unnatural. A list column is defined as having two children, one for the offsets and one for the data. Although an important part of the definition, the offsets are in other ways a detail that an end-user of arrow data typically does not have to think about. In this instance, the problem that arises is that enforcing a 1-1 mapping between child `column_metadata` objects and child columns means that a `column_metadata` instance must be created for the offsets, even though that column will never have a name or children. We should consider changing the definition for lists to only require a single child metadata for the data.

More discussion of this may be found in https://github.com/rapidsai/cudf/pull/16548#discussion_r1715939219 and https://github.com/rapidsai/cudf/issues/16069.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.