[VL] VeloxMemoryPool error
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
```
23/07/29 11:57:55 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
23/07/29 11:57:55 INFO YarnCoarseGrainedExecutorBackend: Got assigned task 4520
23/07/29 11:57:55 ERROR Inbox: Ignoring error
java.util.concurrent.RejectedExecutionException: Task org.apache.spark.executor.Executor$TaskRunner@6fe7e275 rejected from java.util.concurrent.ThreadPoolExecutor@4d543878[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 754]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at org.apache.spark.executor.Executor.launchTask(Executor.scala:270)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:183)
at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
23/07/29 11:57:55 INFO MemoryStore: MemoryStore cleared
23/07/29 11:57:55 INFO BlockManager: BlockManager stopped
WARNING: Logging before InitGoogleLogging() is written to STDERR
E20230729 11:57:55.618413 481318 Exceptions.h:68] Line: /opt/gluten/cpp/velox/memory/VeloxMemoryPool.cc:88, Function:~VeloxMemoryPool, Expression: 0 == remainingBytes (0 vs. 1589248) Memory pool op.1.0.0.FilterProject should be destroyed only after all allocated memory has been freed. Remaining bytes allocated: 1589248, cumulative bytes allocated: 31457280, number of allocations: 882, Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (0 vs. 1589248) Memory pool op.1.0.0.FilterProject should be destroyed only after all allocated memory has been freed. Remaining bytes allocated: 1589248, cumulative bytes allocated: 31457280, number of allocations: 882
Retriable: False
Expression: 0 == remainingBytes
Function: ~VeloxMemoryPool
File: /opt/gluten/cpp/velox/memory/VeloxMemoryPool.cc
Line: 88
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN6gluten15VeloxMemoryPoolD1Ev
# 4 _ZN8facebook5velox4exec4TaskD2Ev
# 5 _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv
# 6 _ZN6gluten24WholeStageResultIteratorD1Ev
# 7 _ZN6gluten35WholeStageResultIteratorMiddleStageD0Ev
# 8 _ZNSt8__detail16_Hashtable_allocISaINS_10_Hash_nodeISt4pairIKlSt10shared_ptrIN6gluten14ResultIteratorEEELb0EEEEE18_M_deallocate_nodeEPS9_.isra.0
# 9 _ZNSt10_HashtableIlSt4pairIKlSt10shared_ptrIN6gluten14ResultIteratorEEESaIS6_ENSt8__detail10_Select1stESt8equal_toIlESt4hashIlENS8_18_Mod_range_hashingENS8_20_Default_ranged_hashENS8_20_Prime_rehash_policyENS8_17_Hashtable_traitsILb0ELb0ELb1EEEE5clearEv
# 10 JNI_OnUnload
# 11 Java_java_lang_ClassLoader_00024NativeLibrary_unload
# 12 0x00007fa08d017a14
```
Contributor guide
Research direction
Start with the reported failure at cpp/velox/memory/VeloxMemoryPool.cc:88 and trace the destruction path through velox::exec::Task, WholeStageResultIterator, and JNI_OnUnload. Compare it with the executor shutdown messages and determine why 1,589,248 bytes remain allocated. Done should include a confirmed explanation and a reproducible validation that the shutdown no longer raises this VeloxRuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, scala
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100