[FEA] [Multi-GPU Polars] Avoid GPU Requirement on Client During IR Lowering
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Avoid GPU Requirement on Client During IR Lowering
### Summary
The current design may require a GPU on the client during the IR lowering stage. Ideally, the client should not require a GPU.
### Problem
IR lowering (`lower_ir_graph`) is currently performed on the client, which can trigger GPU usage. This introduces an implicit requirement that the client machine has a GPU, even in distributed setups where all actual execution happens on workers.
### Proposed Direction
Move IR lowering from the client to the workers.
### Challenges
A naive approach, where each worker independently runs `lower_ir_graph`, would lead to redundant work:
* All workers would sample the same Parquet metadata and row groups.
* This is inefficient and may lead to unnecessary overhead.
cc. @rjzamora, @TomAugspurger
Contributor guide
Assessment
This issue has not been assessed yet.