apache / apache/gluten

[VL] Spark task fails with VeloxRuntime Error

Open
#10,105 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

I was using gluten for last 4 months. Today, I pulled the latest gluten branch and rebuilt the jar with changes, suddenly the spark tasks started to fail with the following exception

`Job aborted due to stage failure: Task 43 in stage 0.0 failed 4 times, most recent failure: Lost task 43.3 in stage 0.0 (TID 153) (10.193.14.135 executor 1): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Empty Optional cannot be unwrapped
Retriable: False
Function: operator()
File: /home/ubuntu/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 571
Stack trace:
0 _ZN8facebook5velox7process10StackTraceC1Ei
1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
3 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
4 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
`

### Gluten version

main branch

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

Ubuntu 22.04

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with Gluten's main branch, Spark 3.5.x, the VL backend, and Ubuntu 22.04, collecting the missing Spark configurations and input query. Then inspect Velox's Driver.cpp around line 571 and the TableScan operator path for the empty optional; done means the reported task no longer fails and a focused regression test or reproducible case covers it.

Written by the indexing model from the issue text.

Assessment

Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.