NVIDIA / NVIDIA/cudf

[BUG] `to_cupy` and `to_numpy` failures on extension dtypes

Open
#23,723 1 comment 0 reactions 0 assignees View on GitHub
? - Needs Triage bug
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

The following test exposes a few different bugs in `to_cupy`/`to_numpy` for extension dtypes for both `Series` and `DataFrame`. I would expect this test to pass for all cases.

```python
import numpy as np
import cudf
import pytest

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
cudf_res = getattr(x, method)(dtype=dtype)
assert cudf_res.dtype == pd_res.dtype
if method == "to_cupy":
cudf_res = cudf_res.get()
np.testing.assert_array_equal(cudf_res, pd_res)
```

And the output

Output

```
============================= test session starts ==============================
platform linux -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/jcristharif/Code/cuml
configfile: pyproject.toml
plugins: xdist-3.8.0, benchmark-5.2.3, cov-7.1.0, hypothesis-6.165.3
collected 8 items

test.py FFFF.FFF [100%]

=================================== FAILURES ===================================
__________ test_to_numpy_or_cupy_extension_types[to_cupy-None-False] ___________

frame = False, dtype = None, method = 'to_cupy'

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
> cudf_res = getattr(x, method)(dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/single_column_frame.py:176: in to_cupy
.to_cupy(dtype=dtype, copy=copy, na_value=na_value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:876: in to_cupy
and cupy.can_cast(col.dtype, final_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

from_ = Int32Dtype(), to = Int32Dtype(), casting = 'safe'

def can_cast(from_, to, casting='safe'):
"""Returns True if cast between data types can occur according to the
casting rule. If from is a scalar or array scalar, also returns True if the
scalar value can be cast without overflow or truncation to an integer.

.. seealso:: :func:`numpy.can_cast`
"""
from_ = from_.dtype if isinstance(from_, ndarray) else from_
> return _numpy.can_cast(from_, to, casting=casting)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E TypeError: Cannot interpret 'Int32Dtype()' as a data type

../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cupy/__init__.py:352: TypeError
___________ test_to_numpy_or_cupy_extension_types[to_cupy-None-True] ___________

frame = True, dtype = None, method = 'to_cupy'

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
> cudf_res = getattr(x, method)(dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:876: in to_cupy
and cupy.can_cast(col.dtype, final_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

from_ = Int32Dtype(), to = Int32Dtype(), casting = 'safe'

def can_cast(from_, to, casting='safe'):
"""Returns True if cast between data types can occur according to the
casting rule. If from is a scalar or array scalar, also returns True if the
scalar value can be cast without overflow or truncation to an integer.

.. seealso:: :func:`numpy.can_cast`
"""
from_ = from_.dtype if isinstance(from_, ndarray) else from_
> return _numpy.can_cast(from_, to, casting=casting)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E TypeError: Cannot interpret 'Int32Dtype()' as a data type

../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cupy/__init__.py:352: TypeError
_________ test_to_numpy_or_cupy_extension_types[to_cupy-float32-False] _________

frame = False, dtype = 'float32', method = 'to_cupy'

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
> cudf_res = getattr(x, method)(dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/single_column_frame.py:176: in to_cupy
.to_cupy(dtype=dtype, copy=copy, na_value=na_value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:876: in to_cupy
and cupy.can_cast(col.dtype, final_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

from_ = Int32Dtype(), to = 'float32', casting = 'safe'

def can_cast(from_, to, casting='safe'):
"""Returns True if cast between data types can occur according to the
casting rule. If from is a scalar or array scalar, also returns True if the
scalar value can be cast without overflow or truncation to an integer.

.. seealso:: :func:`numpy.can_cast`
"""
from_ = from_.dtype if isinstance(from_, ndarray) else from_
> return _numpy.can_cast(from_, to, casting=casting)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E TypeError: did not understand one of the types; 'None' not accepted

../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cupy/__init__.py:352: TypeError
_________ test_to_numpy_or_cupy_extension_types[to_cupy-float32-True] __________

frame = True, dtype = 'float32', method = 'to_cupy'

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
> cudf_res = getattr(x, method)(dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:876: in to_cupy
and cupy.can_cast(col.dtype, final_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

from_ = Int32Dtype(), to = 'float32', casting = 'safe'

def can_cast(from_, to, casting='safe'):
"""Returns True if cast between data types can occur according to the
casting rule. If from is a scalar or array scalar, also returns True if the
scalar value can be cast without overflow or truncation to an integer.

.. seealso:: :func:`numpy.can_cast`
"""
from_ = from_.dtype if isinstance(from_, ndarray) else from_
> return _numpy.can_cast(from_, to, casting=casting)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E TypeError: did not understand one of the types; 'None' not accepted

../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cupy/__init__.py:352: TypeError
__________ test_to_numpy_or_cupy_extension_types[to_numpy-None-True] ___________

frame = True, dtype = None, method = 'to_numpy'

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
cudf_res = getattr(x, method)(dtype=dtype)
> assert cudf_res.dtype == pd_res.dtype
E AssertionError: assert dtype('int32') == dtype('float64')
E + where dtype('int32') = array([[ 1],\n [ 2],\n [-2147483648]], dtype=int32).dtype
E + and dtype('float64') = array([[ 1.],\n [ 2.],\n [nan]]).dtype

test.py:16: AssertionError
________ test_to_numpy_or_cupy_extension_types[to_numpy-float32-False] _________

frame = False, dtype = 'float32', method = 'to_numpy'

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
> cudf_res = getattr(x, method)(dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:929: in to_numpy
return self._to_array(
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:759: in _to_array
return to_array(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

col =
[
1,
2,
null
]
dtype: Int32
to_dtype = 'float32'

def to_array(
col: ColumnBase, to_dtype: Dtype | None
) -> cupy.ndarray | np.ndarray:
if (
col.has_nulls()
and dtype is not None
and not is_string_dtype(dtype)
and na_value is no_default
):
> raise ValueError(
f"cannot convert to '{dtype}'-dtype NumPy array "
"with missing values. Specify an appropriate 'na_value' "
"for this dtype."
)
E ValueError: cannot convert to 'float32'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.

../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:645: ValueError
_________ test_to_numpy_or_cupy_extension_types[to_numpy-float32-True] _________

frame = True, dtype = 'float32', method = 'to_numpy'

@pytest.mark.parametrize("frame", [False, True])
@pytest.mark.parametrize("dtype", [None, "float32"])
@pytest.mark.parametrize("method", ["to_cupy", "to_numpy"])
def test_to_numpy_or_cupy_extension_types(frame, dtype, method):
x = cudf.Series([1, 2, None], dtype="Int32")
if frame:
x = x.to_frame()

pd_res = x.to_pandas().to_numpy(dtype=dtype)
> cudf_res = getattr(x, method)(dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

test.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:929: in to_numpy
return self._to_array(
../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:772: in _to_array
matrix[:, i] = to_array(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

col =
[
1,
2,
null
]
dtype: Int32
to_dtype = 'float32'

def to_array(
col: ColumnBase, to_dtype: Dtype | None
) -> cupy.ndarray | np.ndarray:
if (
col.has_nulls()
and dtype is not None
and not is_string_dtype(dtype)
and na_value is no_default
):
> raise ValueError(
f"cannot convert to '{dtype}'-dtype NumPy array "
"with missing values. Specify an appropriate 'na_value' "
"for this dtype."
)
E ValueError: cannot convert to 'float32'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.

../../miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:645: ValueError
=============================== warnings summary ===============================
test.py::test_to_numpy_or_cupy_extension_types[to_numpy-None-True]
/home/jcristharif/miniforge3/envs/cuml-dev/lib/python3.14/site-packages/cudf/core/frame.py:772: RuntimeWarning: invalid value encountered in cast
matrix[:, i] = to_array(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test.py::test_to_numpy_or_cupy_extension_types[to_cupy-None-False] - T...
FAILED test.py::test_to_numpy_or_cupy_extension_types[to_cupy-None-True] - Ty...
FAILED test.py::test_to_numpy_or_cupy_extension_types[to_cupy-float32-False]
FAILED test.py::test_to_numpy_or_cupy_extension_types[to_cupy-float32-True]
FAILED test.py::test_to_numpy_or_cupy_extension_types[to_numpy-None-True] - A...
FAILED test.py::test_to_numpy_or_cupy_extension_types[to_numpy-float32-False]
FAILED test.py::test_to_numpy_or_cupy_extension_types[to_numpy-float32-True]
==================== 7 failed, 1 passed, 1 warning in 1.32s ====================
```

Contributor guide

Open the contributing guide

Research direction

Start in cudf/core/frame.py and cudf/core/single_column_frame.py, following the to_numpy and to_cupy entry points shown in the traceback. Reproduce the eight cases in test.py, then verify that Series and DataFrame conversions for Int32 with missing values match the pandas results for both dtype settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.