[BUG] Incorrect proxying of functions with no matching fast counterpart in cudf.pandas
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Functions in the pandas source tree which do not have a matching counterpart in the cudf source tree are proxied with a `FunctionProxy` object whose `_fsproxy_fast` attribute is an `_Unusable` object.
Unfortunately, although accessing an `_Unusuable` object in a fast-slow chained method call fails, it does so too late and already provokes slow-to-fast and fast-to-slow copies. This ends up breaking the link between the fast and slow types inside a proxied object.
This raises its head particularly in the pandas test suite where there are functions that are used to parameterise over (for example) `iloc` vs `loc` indexing, like `pandas._testing.iloc`.
To see the problem consider the following:
```python
import cudf.pandas
cudf.pandas.install()
import pandas as pd
s = pd.Series(range(10))
s._fsproxy_state # => FAST
# pd._testing.iloc has no matching fast counterpart, so this function-call will provoke
# a fast to slow copy
indexer = pd._testing.iloc(s)
s._fsproxy_state # => SLOW
# We want setitem to keep the object as slow,
# but this is a `_FastSlowAttribute` so it provokes (if it can) a slow-to-fast copy
getattr(indexer, "__setitem__")
s._fsproxy_state # => FAST
# Now we are in an inconsistent state.
```
In `_transform_arg` we have a carveout early exit if the fast or slow attribute we're asking for is `_Unusable`, but not if it is an instance of `_Unusable`.
This patch helps a bit:
```patch
diff --git a/python/cudf/cudf/pandas/fast_slow_proxy.py b/python/cudf/cudf/pandas/fast_slow_proxy.py
index e811ba1351..9d07d236bb 100644
--- a/python/cudf/cudf/pandas/fast_slow_proxy.py
+++ b/python/cudf/cudf/pandas/fast_slow_proxy.py
@@ -915,7 +915,7 @@ def _transform_arg(
if isinstance(arg, (_FastSlowProxy, _FastSlowProxyMeta, _FunctionProxy)):
typ = getattr(arg, attribute_name)
- if typ is _Unusable:
+ if typ is _Unusable or isinstance(typ, _Unusable):
raise Exception("Cannot transform _Unusable")
return typ
elif isinstance(arg, types.ModuleType) and attribute_name in arg.__dict__:
```
But is observed to cause the pandas test suite run to take significantly longer (indicating, probably, more fast-to-slow transfers than necessary).
Note that this change works for `pd._testing.iloc` but _not_ `pd._testing.setitem` which is just the identity function, since wrapping the identity function produces a new function which is _not_ the identity.
Contributor guide
Assessment
This issue has not been assessed yet.