NVIDIA / NVIDIA/cudf

[FEA] [DISCUSSION] Replace `column_metadata` struct with `ArrowSchema`

Open
#19,244 2 comments 2 reactions 0 assignees View on GitHub
feature request libcudf pylibcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
In cudf's Arrow interop code we use [a custom `column_metadata` class](https://github.com/rapidsai/cudf/blob/cee6214397d85fdc7fc2ae74ff6ed8d13398d10f/cpp/include/cudf/interop.hpp#L119) to forward along additional information needed to convert cudf data into Arrow data. At present, the primary role of this object is to provide a way to assign field names to structs because libcudf does not have struct field names but they are part of the Arrow specification (more details in https://github.com/rapidsai/cudf/issues/19215). However, there may be other data that is also required for specific conversions that we have not encountered yet.

The Arrow Python Capsule interface has the concept of [schema requests](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) by which users can request conversion of the implementing type to be coerced to a particular Arrow schema during the export process. Currently we do not support that in cuDF Python, and the underlying libcudf APIs do not support a requesting schema. However, we de facto support a limited subset of this functionality by accepting `column_metadata` objects instead, specifically for the purpose of assigning struct field names. This choice is an artifact of an older implementation that predates the Arrow C Data interface. It is less flexible than the alternative and adds a translation layer from any consumer that may wish to make such requests using the Arrow specification. As such, moving towards using `ArrowSchema` directly seems like a better long-term outcome.

**Describe the solution you'd like**
I propose that we implement new versions of our `to_arrow*` conversion APIs that accept `ArrowSchema` objects instead of `column_metadata` objects. That would allow us the flexibility to support any conversion that we can that also fits within the Arrow specification, rather than having to continue extending our own storage medium for this. It would also allow us to interface cleanly with consumers that already know how to produce `ArrowSchema` objects by allowing them to avoid `column_metadata` translation.

**Describe alternatives you've considered**
While `ArrowSchema` objects are fairly lightweight since they do not carry around any data buffers, they are still more complex than `column_metadata`, so there is some additional cost to using them instead. Moreover, there is some additional complexity since we will need to use nanoarrow functions (or equivalent) to extract data. However, I think that this cost is worth paying for the flexibility and interoperability gains discussed above. Moreover, it saves us from having to resolve ambiguities like those in https://github.com/rapidsai/cudf/issues/16600 ourselves since we will be sticking to the well-established Arrow specification and preexisting community-driven design decisions.

**Additional context**
I am aware of at least two use cases that `column_metadata` does not currently support that we might need (and which `ArrowSchema` does):
- Timezone support, see #19155
- Offline discussions around decimal scale. libcudf's fixed point types only have a scale and always assume max precision, while [the Arrow specification also includes a scale in the schema for decimal data](https://github.com/apache/arrow/blob/ed13cedd8bf7ddc06db152f97e68d86c2c37e949/format/Schema.fbs#L231).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.