Track and avoid repeated fallback issues in cudf.pandas

One class of issues that we frequently see as a performance bottleneck in cudf.pandas is that some workflow will hammer a pandas API that cudf does not support and we will do a large number of memory transfers in the fallback process. When usage of that API is high enough, the memory transfers alone become the dominant factor in the performance of the workflow. We have improved this situation incrementally by a few approaches such as poisoning specific proxy methods to prevent trying the fast operation, but ultimately that always leaves us open to new blind spots.

I propose that we switch to a more principled approach. We could record all cases where an API falls back during a given workflow, and on future calls to the same API we could prevent even trying the operation on the device, preempting an unnecessary H2D2D round trip. I see some challenges with this approach, which I enumerate below, but if we can find a way to do this it would save us a lot of overhead.

The big challenges I see are around how we would track function calls. Would we include all arguments? The arguments could be cudf/pandas objects, so would we also have to hash their contents to determine that we're making the same call? I suspect that would lead to too few cache hits, but then we have to decide what constitutes suitable granularity. Maybe you skip all the data containers but include only the primitive type parameters, which boil down to being flags. If we did need to include any non-primitive types, we would also need to make sure to use weakrefs or similar to avoid the cache keeping objects alive since we don't want to extend lifetimes just for this purpose.