apache / apache/gluten

[VL] GPU code shouldn't be running on CPU node when cudf is enabled

Open
#11,828 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Currently, Gluten uses the configuration `spark.gluten.sql.columnar.cudf=true` to enable GPU support. When running on a hybrid cluster, CPU nodes will fail if this configuration is set. We should disable the GPU code by checking whether the node has the CUDA runtime and available devices.

Another issue that needs to be resolved is that we currently pass the cudf configurations `spark.gluten.sql.columnar.cudf` and `spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan` to the session configuration at runtime for the WholeStageResultIterator. However, these are immutable configurations. We should remove them from the runtime session configuration and instead set their values from the backend configuration.

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with the Velox backend code involved in WholeStageResultIterator and trace how cudf settings enter the runtime session configuration. Then inspect the existing backend configuration and CUDA availability checks, including hybrid CPU/GPU execution. Done means CPU nodes avoid GPU code when CUDA devices are unavailable and immutable cudf settings come from backend configuration rather than runtime session configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.