apache / apache/gluten

Memory leak issue

Open
#8,362 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Exception:
```
E20241226 23:24:39.978191 324668 VeloxMemoryManager.cc:401] Failed to release Velox memory manager after 43350ms as there are still outstanding memory resources.
E20241226 23:24:39.978293 324668 MemoryPool.cpp:442] [MEM] Memory leak (Used memory): Memory Pool[default_leaf LEAF root[root] parent[root] MALLOC track-usage thread-safe]
E20241226 23:24:39.979336 324668 Exceptions.h:66] Line: /work/cpp/velox/memory/VeloxMemoryManager.cc:102, Function:removePool, Expression: pool->reservedBytes() == 0 (1619001344 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (1619001344 vs. 0)
Retriable: False
Expression: pool->reservedBytes() == 0
Function: removePool
File: /work/cpp/velox/memory/VeloxMemoryManager.cc
Line: 102
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN6gluten20ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
# 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
# 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
# 6 _ZN6gluten18VeloxMemoryManagerD1Ev
# 7 _ZN6gluten18VeloxMemoryManagerD0Ev
# 8 _ZN6gluten13MemoryManager7releaseEPS0_
# 9 Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 10 0x00007ff6b0bc7be4

.
Driver stacktrace:
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:45)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:341)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:198)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:83)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:55)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:198)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:193)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:207)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 152 in stage 9.0 failed 4 times, most recent failure: Lost task 152.3 in stage 9.0 (TID 24199, 10.10.1.127, executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container from a bad node: container_e2320_1734105600630_7003_01_000015 on host: 10.10.1.127. Exit status: 134. Diagnostics: :61)
at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.wrapForeignAllocation(BaseAllocator.java:241)
at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.wrapForeignAllocation(RootAllocator.java:29)
at org.apache.arrow.c.ReferenceCountedArrowArray.unsafeAssociateAllocation(ReferenceCountedArrowArray.java:67)
at org.apache.arrow.c.BufferImportTypeVisitor.importBuffer(BufferImportTypeVisitor.java:99)
at org.apache.arrow.c.BufferImportTypeVisitor.importData(BufferImportTypeVisitor.java:120)
at org.apache.arrow.c.BufferImportTypeVisitor.visit(BufferImportTypeVisitor.java:208)
at org.apache.arrow.c.BufferImportTypeVisitor.visit(BufferImportTypeVisitor.java:61)
at org.apache.gluten.shaded.org.apache.arrow.vector.types.pojo.ArrowType$Utf8.accept(ArrowType.java:739)
at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:126)
... 138 more

```

### Spark version

Spark-3.5.x

### Spark configurations

key|value
-- | --
spark.gluten.memory.conservative.task.offHeap.size.in.bytes | 1640438897
spark.executor.memory|45g
spark.executor.cores|18
spark.gluten.memory.dynamic.offHeap.sizing.enabled | false
spark.gluten.memory.offHeap.size.in.bytes | 59055800320
spark.gluten.memory.task.offHeap.size.in.bytes | 3280877795
spark.gluten.memoryOverhead.size.in.bytes | 15032385536

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.