apache / apache/gluten

[VL] Deadlock when memory arbitration in TableScan preload

Open
#9,846 4 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

If reclaim happened in TableScan preload's IO thread during `TableScan::getOuput()` waiting async preload, reclaimTask will wait `TableScan::getOuput()` until timeout.

Related to https://github.com/apache/incubator-gluten/issues/7800 and https://github.com/apache/incubator-gluten/pull/7799.

### Gluten version

1.3

### Spark version

3.5

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Memory reclaim failed to wait for task Gluten_Stage_2_TID_12269_VTID_1 to pause after 1h 0m 0s with max timeout 1h 0m 0s
Retriable: False
Context: Split Hive: ...
File: /tmp/build/src/dev/meituan/src/velox/velox/exec/Task.cpp
Line: 3161
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4exec4Task15MemoryReclaimer11reclaimTaskERKSt10shared_ptrIS2_EmmRNS0_6memory15MemoryReclaimer5StatsE
# 4 _ZN8facebook5velox4exec4Task15MemoryReclaimer7reclaimEPNS0_6memory10MemoryPoolEmmRNS4_15MemoryReclaimer5StatsE
# 5 _ZN8facebook5velox6memory15MemoryReclaimer7reclaimEPNS1_10MemoryPoolEmmRNS2_5StatsE
# 6 _ZN6gluten20ListenableArbitrator14shrinkCapacityEmbb
# 7 _ZN6gluten24WholeStageResultIterator14spillFixedSizeEl
# 8 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill
# 9 0x00007fd0235a2a34
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.