NVIDIA / NVIDIA/cudf

[FEA] [Multi-GPU Polars] Avoid GPU Requirement on Client During IR Lowering

Open
#21,841 3 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request rapids-mpf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Avoid GPU Requirement on Client During IR Lowering

### Summary

The current design may require a GPU on the client during the IR lowering stage. Ideally, the client should not require a GPU.

### Problem

IR lowering (`lower_ir_graph`) is currently performed on the client, which can trigger GPU usage. This introduces an implicit requirement that the client machine has a GPU, even in distributed setups where all actual execution happens on workers.

### Proposed Direction

Move IR lowering from the client to the workers.

### Challenges

A naive approach, where each worker independently runs `lower_ir_graph`, would lead to redundant work:

* All workers would sample the same Parquet metadata and row groups.
* This is inefficient and may lead to unnecessary overhead.

cc. @rjzamora, @TomAugspurger

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.