RFC Automatically choose best namespace/device within an estimator

[!WARNING]
This issue is not yet ready for a PR. If you are interested in contributing to scikit-learn, please have a look at our contributing guidelines, and in particular the sections for new contributors and the "Needs triage" label.

This is a RFC related to array API.

The central question is: who knows best when to run a computation on the CPU vs on a GPU/accelerated hardware?

So far we've been assuming that the user will know when to move data from one device to the other. As a result we need something like the MoveTo transformer. There are probably other things that result from this assumption, but I can't think of them right now.

Inspired by using sentence-transformers and realising that it automatically uses my GPU when I run the code on a machine with a GPU, the MPS device of my MacBook and otherwise stays on the CPU. This was a cool experience because I didn't have to go and edit my code. The input to sentence-transformers was always a Numpy array but it knew that if there was a CUDA device it made sense to use it. From talking about this with Olivier I learnt that TabICL also does this (the processing starts on the CPU but it then moves to the GPU).

Roughly at the same time I investigated how long it takes to move different amounts of data from CPU memory to GPU memory (and back). Some results and the script used to measure it https://gist.github.com/betatim/f4128b961525eda49df9b76c7c339388 (ignore the "predicted" column, it isn't that useful). The main thing I took away from this is that even fairly large datasets take about half a second. Combined with my assumption that the work that happens in the estimator takes at least a second or two and sometimes much longer it made me think "if this is true then moving data is 'cheap'". I'm not 100% there isn't a mistake or logic error here. Let me know what you think.

The result from that discussion and the transfer time experiment is this RFC.

The proposal is that an estimator is allowed to move from one namespace/device to another if it thinks that it makes sense to do so. So for example you feed a numpy array to Ridge which then decides to move to pytorch and the GPU to do the computation. This behaviour would be controlled by a global setting (opt-in at the start). Each estimator probably also needs to have some public API to control its behaviour.

It would remove the need for having a MoveTo transformer and free the user from having to think about this/know when best to transition from one namespace/device to another. It would also make it easier to have code for a pipeline that works on a machine with and without a GPU - no need to edit the code.

We already kinda do this for some (parts of some) estimators. Mostly we do it because there is no array API function to do what we want to do, but there is a Numpy one. Internally the estimator moves the data to numpy, performs the computation and then moves the result back (if needed).

We'd have to have some "order of precedence" function that knows how to check if cupy, pytorch, dpnp are installed, if there is an accelerator present that is supported by the installed library and then decided what to do. I think in practice this is not that complicated as most machines will only have one type of accelerator and one of these libraries installed. I think sentence-transformers uses PyTorch and then: CUDA > MPS > CPU (for example here and then used here)

Overall I think this would be a nice UX for users if it doesn't become "too magical".

What do people think?

Start by reading this RFC and the linked MoveTo transformer issue, then review the discussion about array namespaces, devices, and estimator behavior. The work is not ready for a PR; done means reaching agreement on whether automatic selection is appropriate and defining the global setting, estimator API, and precedence rules.