[VL] Non-spillable state during Velox OrderBy out-stage
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
```
4/01/24 09:19:17 WARN [task-result-getter-2] scheduler.TaskSetManager: Lost task 2286.0 in stage 3.0 (TID 188286) (monarch-dev-015-20221117-dpp-worker-dev-0a027acd.ec2.pin220.com executor 385): io.glutenproject.exception.GlutenException: java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: OrderBy, plan node ID: 1]: Error during calling Java code from native code: io.glutenproject.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8388608, granted: 0. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application.
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=50.0 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=12.5 GiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=6.3 GiB
Memory consumer stats:
Task.188286: Current used bytes: 12.5 GiB, peak bytes: N/A
+- org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@73070ba7: Current used bytes: 12.4 GiB, peak bytes: N/A
\- Gluten.Tree.314: Current used bytes: 128.0 MiB, peak bytes: 12.4 GiB
\- root.314: Current used bytes: 128.0 MiB, peak bytes: 12.4 GiB
+- WholeStageIterator.314: Current used bytes: 112.0 MiB, peak bytes: 9.8 GiB
| \- single: Current used bytes: 112.0 MiB, peak bytes: 9.5 GiB
| +- task.Gluten_Stage_3_TID_188286: Current used bytes: 112.0 MiB, peak bytes: 9.5 GiB
| | +- node.1: Current used bytes: 112.0 MiB, peak bytes: 9.5 GiB
| | | \- op.1.0.0.OrderBy: Current used bytes: 111.8 MiB, peak bytes: 9.1 GiB
| | +- node.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.0.0.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- node.2: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- op.2.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- WholeStageIterator_default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ColumnarToRow.0: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
| \- single: Current used bytes: 1024.0 KiB, peak bytes: 2.0 MiB
| \- ColumnarToRow_default_leaf: Current used bytes: 1024.0 KiB, peak bytes: 1792.0 KiB
+- ArrowContextInstance.315: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
+- ShuffleReader.0: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| \- single: Current used bytes: 0.0 B, peak bytes: 1024.0 KiB
| \- ShuffleReader_default_leaf: Current used bytes: 0.0 B, peak bytes: 576.0 KiB
+- OverAcquire.DummyTarget.1106: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
+- OverAcquire.DummyTarget.1102: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
\- OverAcquire.DummyTarget.1098: Current used bytes: 0.0 B, peak bytes: 2.9 GiB
[7:37 AM] Yang, Binwei
at io.glutenproject.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:90)
at io.glutenproject.memory.nmm.ManagedReservationListener.reserve(ManagedReservationListener.java:43)
at io.glutenproject.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at io.glutenproject.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:65)
at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
at io.glutenproject.utils.IteratorCompleter.hasNext(Iterators.scala:69)
at io.glutenproject.utils.PayloadCloser.hasNext(Iterators.scala:35)
at io.glutenproject.utils.PipelineTimeAccumulator.hasNext(Iterators.scala:98)
at io.glutenproject.execution.VeloxColumnarToRowExec$$anon$1.hasNext(VeloxColumnarToRowExec.scala:107)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.TraversableOnce$FlattenOps$$anon$2.hasNext(TraversableOnce.scala:469)
at io.glutenproject.utils.InvocationFlowProtection.hasNext(Iterators.scala:135)
at io.glutenproject.utils.IteratorCompleter.hasNext(Iterators.scala:69)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:225)
at org.apache.spark.sql.execution.SortExec.$anonfun$doExecute$1(SortExec.scala:119)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:984)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:984)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:338)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Retriable: False
Function: runInternal
File: /home/binweiyang/gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 582
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE.cold
# 4 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 5 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 6 _ZN6gluten24WholeStageResultIterator4nextEv
# 7 Java_io_glutenproject_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 8 0x00007f36664d4928
at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
at io.glutenproject.utils.IteratorCompleter.hasNext(Iterators.scala:69)
at io.glutenproject.utils.PayloadCloser.hasNext(Iterators.scala:35)
at io.glutenproject.utils.PipelineTimeAccumulator.hasNext(Iterators.scala:98)
at io.glutenproject.execution.VeloxColumnarToRowExec$$anon$1.hasNext(VeloxColumnarToRowExec.scala:107)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.TraversableOnce$FlattenOps$$anon$2.hasNext(TraversableOnce.scala:469)
at io.glutenproject.utils.InvocationFlowProtection.hasNext(Iterators.scala:135)
at io.glutenproject.utils.IteratorCompleter.hasNext(Iterators.scala:69)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:225)
at org.apache.spark.sql.execution.SortExec.$anonfun$doExecute$1(SortExec.scala:119)
```
Issue may relate to https://github.com/oap-project/gluten/issues/4275#issuecomment-1888552938
Contributor guide
Assessment
This issue has not been assessed yet.