[Story] Support partial on-device query execution for cudf-polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Currently, the translation of a polars query into something that executes on device is an all-or-nothing affair. If we determine that we cannot execute the plan, we fail during translation and the usual polars cpu engine executes the query. In a number of scenarios it would be useful to support hybrid execution.
For example, we currently do not support IO from remote IO sources. Polars has a bunch of cloud-interfacing IO routines. If we supported partial plan execution, we could use the polars CPU side to produce the input dataframe (after query optimisations), transfer and run rest of the query on the GPU, and ship back the result. We could also produce only a partial query from the root on the GPU and then hand off to the CPU for (say) string matching if we don't support some regex feature.
Implementing subtree evaluation (back to a source node) is doable with our current infrastructure. We would keep track of the nodes we've been translating, and then if we hit a failure in translation unwind and just set the child node as the translated one.
To replace trees where the inner children are evaluated on the CPU, we need a small extension on the polars side. Right now, we can replace a node with a `PythonScan` node (which models a source: `IO () -> DataFrame`), to keep children, we need to also be able to replace a node with a `OpaquePython` `MapFunction` (which models a `DataFrame -> DataFrame` translation). This would avoid us needing to recreate a polars node, we could just replace the node that acts on the input node with our `OpaquePython` node.
Contributor guide
Assessment
This issue has not been assessed yet.