NVIDIA / NVIDIA/cudf

[FEA] Consider default nullability setting when converting cudf data to arrow C Data interface

Open
#16,621 4 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
[The implementation of `to_arrow_schema.cpp` sets the nullability of a column in the schema based on whether the cudf column contains nulls](https://github.com/rapidsai/cudf/blob/branch-24.10/cpp/src/interop/to_arrow_schema.cpp#L221). This behavior differs from the [default behavior of Arrow, where schema are created as nullable by default](https://arrow.apache.org/docs/cpp/api/datatype.html#_CPPv46schemaNSt16initializer_listINSt4pairINSt6stringENSt10shared_ptrI8DataTypeEEEEEENSt10shared_ptrIK16KeyValueMetadataEE). However, the behavior doesn't seem quite so straightforward in all cases. In pylibcudf testing, [we currently ignore field nullability by default](https://github.com/rapidsai/cudf/blob/branch-24.10/python/pylibcudf/pylibcudf/tests/common/utils.py#L47) because this proved problematic in https://github.com/rapidsai/cudf/pull/16548, but simply flipping the behavior of the above line to always make columns nullable also did not resolve the discrepancy (i.e. setting `check_field_nullability=True` causes tests to fail).

**Describe the solution you'd like**
We should determine whether there is a different choice of default that would be more compatible with all of the cases that we need to support.

**Describe alternatives you've considered**
The status quo may also be acceptable. There don't appear to be many cases where nullability of columns containing no nulls is relevant outside of our more stringent tests. However, https://github.com/rapidsai/cudf/pull/16590#pullrequestreview-2248058144 indicates that there may be edge cases where this is important and we might need to adjust for those, hence opening this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.