[Story] Consider attaching name metadata to pylibcudf columns and struct types
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Currently pylibcudf objects are designed to have a 1-1 correspondence with libcudf objects. In libcudf, columns in a table are identified purely by index, as are members in a struct type column. Arrow requires struct type to have a name, however, and essentially all implementations built on Arrow also assume that columns have names. Since columns and struct members are both ordered, libcudf's representation is functionally equivalent to the Arrow representation for all algorithms, but the incomplete bijection means that conversion from the Arrow representation is lossy.
The lossy conversion causes us a number of issues.
- Round-tripping struct columns will fail (#17061)
- Round-tripping any columns will fail if the other type (e.g. a polars DataFrame) requires unique or non-empty column names
- Each higher-level library (cudf classic/cudf.pandas, cudf-polars) needs to track this information on its own
- The capsule interfaces (like `__arrow_c_array__`) cannot be the primary approach for converting pylibcudf objects to corresponding host objects because there is no currently supported way in pylibcudf for assigning those names, and even if there were the calling libraries would not know that they had to pass in that information. That limits our ability to use the capsule interfaces to replace more explicitly conversion APIs.
Our more recent support for the Arrow C Data Interface and the associated data types also uncovers some friction. We have a custom `column_metadata` type that we need to construct and pass down to the `to_arrow*` conversions to get struct element names set on the output, but we do not yet support any easy way to use this to modify the `ArrowSchema` produced by the `to_arrow_schema` functions/methods. As a result, we also cannot easily support the `requested_schema` parameter in the capsule interfaces like `__arrow_c_device_array_`.
**Describe the solution you'd like**
We should consider adding native support for column and struct names to pylibcudf. Some questions that we need to answer:
- Should we use a new wrapper type (e.g. `TableWithNames`) or add this information directly to `Table`?
- Should we make column names required or optional?
- How do we tack this information onto struct types? This is more complicated than column names since we use a single DataType for all types with just a type id to distinguish types. We could follow a similar practice to what we do in libcudf with decimal types where the data type always carries around a scale, but that feels a bit messy and tacked on.
Contributor guide
Assessment
This issue has not been assessed yet.