apache / apache/gluten

[VL] "Spill level 4 exceeds max spill level 3" error

Open
#11,018 9 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

error:
```
W20251102 22:06:55.722939 2682850 HashProbe.cpp:1837] Can't reclaim from hash probe operator, state_[WAIT_FOR_BUILD], nonReclaimableSection_[0], inputSpiller_[nullptr], table_[nullptr], table_ numDistinct[nullptr], op.3.0.0.HashProbe, usage: 1.94MB, node pool reservation: 8.00MB
25/11/02 22:06:55 INFO NativePlanEvaluator: NativePlanEvaluator-5: Spilled 1335885824 / 52953088 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 / 80425779 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 / 80425779 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 7340032 / 80425779 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 / 73085747 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 / 73085747 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 / 73085747 bytes of data.
W20251102 22:07:03.134801 2682850 HashProbe.cpp:1837] Can't reclaim from hash probe operator, state_[WAIT_FOR_BUILD], nonReclaimableSection_[0], inputSpiller_[nullptr], table_[nullptr], table_ numDistinct[nullptr], op.3.0.0.HashProbe, usage: 1.89MB, node pool reservation: 11.00MB
25/11/02 22:07:03 INFO NativePlanEvaluator: NativePlanEvaluator-5: Spilled 1349517312 / 73085747 bytes of data.
E20251102 22:07:03.571292 2682850 Exceptions.h:66] Line: /work/ep/build-velox/build/velox_ep/velox/exec/Spill.cpp:469, Function:SpillPartitionId, Expression: Spill level 4 exceeds max spill level 3, Source: RUNTIME, ErrorCode: INVALID_STATE
E20251102 22:07:03.584228 2682850 Exceptions.h:66] Line: /work/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp:574, Function:operator(), Expression: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Spill level 4 exceeds max spill level 3
Retriable: False
Context: Operator: HashBuild[3] 2
Additional Context: Operator: ValueStream[0] 0
Function: SpillPartitionId
File: /work/ep/build-velox/build/velox_ep/velox/exec/Spill.cpp
Line: 469
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4exec16SpillPartitionIdC1ES2_j
# 4 _ZN8facebook5velox4exec11SpillerBase11finishSpillERSt3mapINS1_16SpillPartitionIdESt10unique_ptrINS1_14SpillPartitionESt14default_deleteIS6_EESt4lessIS4_ESaISt4pairIKS4_S9_EEE
# 5 _ZN8facebook5velox4exec9HashBuild15finishHashBuildEv
# 6 _ZN8facebook5velox4exec9HashBuild19noMoreInputInternalEv
# 7 _ZN8facebook5velox4exec9HashBuild20postHashBuildProcessEv
# 8 _ZN8facebook5velox4exec9HashBuild9isBlockedEPN5folly10SemiFutureINS3_4UnitEEE
# 9 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE0_clEv
# 10 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 11 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 12 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 13 _ZN6gluten24WholeStageResultIterator4nextEv
# 14 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 15 0x00007fab220da0a8

at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:95)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at org.apache.gluten.vectorized.ColumnarBatchInIterator.hasNext(ColumnarBatchInIterator.java:36)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:95)
at org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:122)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at org.apache.gluten.vectorized.ColumnarBatchInIterator.hasNext(ColumnarBatchInIterator.java:36)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:122)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.writer.VeloxUniffleColumnarShuffleWriter.writeImpl(VeloxUniffleColumnarShuffleWriter.java:149)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Begin with velox/exec/Spill.cpp at line 469, then follow SpillPartitionId through SpillerBase::finishSpill and HashBuild::finishHashBuild as shown in the native stack trace. Establish a reproducible workload using the missing Gluten, Spark, configuration, and system details; done means it completes without the INVALID_STATE spill-level error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, scala
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.