[VL] Spark task fails with VeloxRuntime Error
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
I was using gluten for last 4 months. Today, I pulled the latest gluten branch and rebuilt the jar with changes, suddenly the spark tasks started to fail with the following exception
`Job aborted due to stage failure: Task 43 in stage 0.0 failed 4 times, most recent failure: Lost task 43.3 in stage 0.0 (TID 153) (10.193.14.135 executor 1): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Empty Optional cannot be unwrapped
Retriable: False
Function: operator()
File: /home/ubuntu/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 571
Stack trace:
0 _ZN8facebook5velox7process10StackTraceC1Ei
1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
3 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
4 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
`
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
Ubuntu 22.04
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by reproducing the failure with Gluten's main branch, Spark 3.5.x, the VL backend, and Ubuntu 22.04, collecting the missing Spark configurations and input query. Then inspect Velox's Driver.cpp around line 571 and the TableScan operator path for the empty optional; done means the reported task no longer fails and a focused regression test or reproducible case covers it.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100