NVIDIA / NVIDIA/cudf

[BUG] cugraph.Graph.from_pandas_adjacency fails with cudf.pandas if pyarrow.dataset import is removed.

Open
#18,741 0 comments 0 reactions 0 assignees View on GitHub
bug cudf.pandas Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
While working on https://github.com/rapidsai/cudf/pull/18663, making a `pyarrow.dataset`import lazy (i.e. not always importing it when `import cudf`) causes some cuGraph third party integration tests to fail with a `RecursionError`

**Steps/Code to reproduce bug**
When running
```bash
pytest -p cudf.pandas python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_cugraph.py::test_cugraph_from_pandas_adjacency -x
```

without the `import pyarrow.dataset` leads to this `RecursionError` (posting the part that recurses)

```python
File "/cudf/python/cudf/cudf/pandas/fast_slow_proxy.py", line 540, in _fsproxy_slow
self._fsproxy_wrapped = self._fsproxy_fast_to_slow()
~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "conda/envs/cudf-dev/lib/python3.13/site-packages/nvtx/nvtx.py", line 122, in inner
result = func(*args, **kwargs)
File "/cudf/python/cudf/cudf/pandas/fast_slow_proxy.py", line 204, in _fsproxy_fast_to_slow
return fast_to_slow(self._fsproxy_wrapped)
File "cudf/python/cudf/cudf/pandas/_wrappers/pandas.py", line 280, in
fast_to_slow=lambda fast: fast.to_pandas(),
~~~~~~~~~~~~~~^^
File "cudf/python/cudf/cudf/utils/performance_tracking.py", line 51, in wrapper
return func(*args, **kwargs)
File "cudf/python/cudf/cudf/core/series.py", line 2061, in to_pandas
self._column.to_pandas(nullable=nullable, arrow_type=arrow_type),
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cudf/python/cudf/cudf/core/column/string.py", line 6179, in to_pandas
return super().to_pandas(nullable=nullable, arrow_type=arrow_type)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cudf/python/cudf/cudf/core/column/column.py", line 710, in to_pandas
return pd.Index(pa_array.to_pandas())
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
File "conda/envs/cudf-dev/lib/python3.13/site-packages/pandas/core/indexes/base.py", line 485, in __new__
name = maybe_extract_name(name, data, cls)
File "conda/envs/cudf-dev/lib/python3.13/site-packages/pandas/core/indexes/base.py", line 7692, in maybe_extract_name
if name is None and isinstance(obj, (Index, ABCSeries)):
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "conda/envs/cudf-dev/lib/python3.13/site-packages/pandas/core/dtypes/generic.py", line 44, in _instancecheck
return _check(inst) and not isinstance(inst, type)
~~~~~~^^^^^^
File "conda/envs/cudf-dev/lib/python3.13/site-packages/pandas/core/dtypes/generic.py", line 38, in _check
return getattr(inst, attr, "_typ") in comp
~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "python/cudf/cudf/pandas/fast_slow_proxy.py", line 862, in __get__
getattr(instance._fsproxy_slow, self._name),
^^^^^^^^^^^^^^^^^^^^^^
File "python/cudf/cudf/pandas/fast_slow_proxy.py", line 540, in _fsproxy_slow
self._fsproxy_wrapped = self._fsproxy_fast_to_slow()
~~~~~~~~~~~~~~~~~~~~~~~~~~^^
...
RecursionError: maximum recursion depth exceeded
```

A workaround in https://github.com/rapidsai/cudf/pull/18663 is to `import pyarrow.dataset` when defining our pandas proxy objects, but this is a stop gap. I believe the issue relates to the "pandas-shim", described in https://github.com/rapidsai/cudf/issues/14521#issue-2015198786, that pyarrow uses when interacting with pandas. (This does not get imported when `import pyarrow` is called but does when `import pyarrow.dataset` is called)

**Expected behavior**
This should not recurse.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.