[VL] GPU code shouldn't be running on CPU node when cudf is enabled
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Currently, Gluten uses the configuration `spark.gluten.sql.columnar.cudf=true` to enable GPU support. When running on a hybrid cluster, CPU nodes will fail if this configuration is set. We should disable the GPU code by checking whether the node has the CUDA runtime and available devices.
Another issue that needs to be resolved is that we currently pass the cudf configurations `spark.gluten.sql.columnar.cudf` and `spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan` to the session configuration at runtime for the WholeStageResultIterator. However, these are immutable configurations. We should remove them from the runtime session configuration and instead set their values from the backend configuration.
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with the Velox backend code involved in WholeStageResultIterator and trace how cudf settings enter the runtime session configuration. Then inspect the existing backend configuration and CUDA availability checks, including hybrid CPU/GPU execution. Done means CPU nodes avoid GPU code when CUDA devices are unavailable and immutable cudf settings come from backend configuration rather than runtime session configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100