[VL] Deadlock when memory arbitration in TableScan preload
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
If reclaim happened in TableScan preload's IO thread during `TableScan::getOuput()` waiting async preload, reclaimTask will wait `TableScan::getOuput()` until timeout.
Related to https://github.com/apache/incubator-gluten/issues/7800 and https://github.com/apache/incubator-gluten/pull/7799.
### Gluten version
1.3
### Spark version
3.5
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Memory reclaim failed to wait for task Gluten_Stage_2_TID_12269_VTID_1 to pause after 1h 0m 0s with max timeout 1h 0m 0s
Retriable: False
Context: Split Hive: ...
File: /tmp/build/src/dev/meituan/src/velox/velox/exec/Task.cpp
Line: 3161
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4exec4Task15MemoryReclaimer11reclaimTaskERKSt10shared_ptrIS2_EmmRNS0_6memory15MemoryReclaimer5StatsE
# 4 _ZN8facebook5velox4exec4Task15MemoryReclaimer7reclaimEPNS0_6memory10MemoryPoolEmmRNS4_15MemoryReclaimer5StatsE
# 5 _ZN8facebook5velox6memory15MemoryReclaimer7reclaimEPNS1_10MemoryPoolEmmRNS2_5StatsE
# 6 _ZN6gluten20ListenableArbitrator14shrinkCapacityEmbb
# 7 _ZN6gluten24WholeStageResultIterator14spillFixedSizeEl
# 8 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill
# 9 0x00007fd0235a2a34
```
Contributor guide
Assessment
This issue has not been assessed yet.