apache / apache/gluten

[FLINK] Memory leak when trying to run nexmark with rocksdb state backend

Open
#11,791 0 comments 2 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

When I run, for example, nexmark q4 with state backend type rocksdb, the job fails. It happens because for some reason task manager drops. Logs of taskmanager just interrupted, which indicates unexpected crash. However, there is some info in stdout:

```bash
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
Created work directory /tmp/velox4j-307006935592892106.
Found required libraries in container velox4j-lib/Linux/amd64.
Copying library velox4j-lib/Linux/amd64/libevent-2.0.so.5 to /tmp/velox4j-307006935592892106/lib/libevent-2.0.so.5...
Copying library velox4j-lib/Linux/amd64/libvelox.so to /tmp/velox4j-307006935592892106/lib/libvelox.so...
Copying library velox4j-lib/Linux/amd64/libcrypto.so.1.1 to /tmp/velox4j-307006935592892106/lib/libcrypto.so.1.1...
Copying library velox4j-lib/Linux/amd64/libfmt.so.9 to /tmp/velox4j-307006935592892106/lib/libfmt.so.9...
Copying library velox4j-lib/Linux/amd64/libaio.so.1 to /tmp/velox4j-307006935592892106/lib/libaio.so.1...
Copying library velox4j-lib/Linux/amd64/libzstd.so.1 to /tmp/velox4j-307006935592892106/lib/libzstd.so.1...
Copying library velox4j-lib/Linux/amd64/libgflags.so.2.2 to /tmp/velox4j-307006935592892106/lib/libgflags.so.2.2...
Copying library velox4j-lib/Linux/amd64/libfolly.so.0.58.0-dev to /tmp/velox4j-307006935592892106/lib/libfolly.so.0.58.0-dev...
Copying library velox4j-lib/Linux/amd64/libssl.so.1.1 to /tmp/velox4j-307006935592892106/lib/libssl.so.1.1...
Copying library velox4j-lib/Linux/amd64/libicuuc.so.66 to /tmp/velox4j-307006935592892106/lib/libicuuc.so.66...
Copying library velox4j-lib/Linux/amd64/libsodium.so.23 to /tmp/velox4j-307006935592892106/lib/libsodium.so.23...
Copying library velox4j-lib/Linux/amd64/libsasl2.so.2 to /tmp/velox4j-307006935592892106/lib/libsasl2.so.2...
Copying library velox4j-lib/Linux/amd64/liblzma.so.5 to /tmp/velox4j-307006935592892106/lib/liblzma.so.5...
Copying library velox4j-lib/Linux/amd64/libboost_program_options.so.1.81.0 to /tmp/velox4j-307006935592892106/lib/libboost_program_options.so.1.81.0...
Copying library velox4j-lib/Linux/amd64/libboost_filesystem.so.1.81.0 to /tmp/velox4j-307006935592892106/lib/libboost_filesystem.so.1.81.0...
Copying library velox4j-lib/Linux/amd64/libdouble-conversion.so.3 to /tmp/velox4j-307006935592892106/lib/libdouble-conversion.so.3...
Copying library velox4j-lib/Linux/amd64/libboost_context.so.1.81.0 to /tmp/velox4j-307006935592892106/lib/libboost_context.so.1.81.0...
Copying library velox4j-lib/Linux/amd64/librdkafka.so.1 to /tmp/velox4j-307006935592892106/lib/librdkafka.so.1...
Copying library velox4j-lib/Linux/amd64/libstdc++.so.6 to /tmp/velox4j-307006935592892106/lib/libstdc++.so.6...
Copying library velox4j-lib/Linux/amd64/libunwind.so.8 to /tmp/velox4j-307006935592892106/lib/libunwind.so.8...
Copying library velox4j-lib/Linux/amd64/libz.so.1 to /tmp/velox4j-307006935592892106/lib/libz.so.1...
Copying library velox4j-lib/Linux/amd64/liblz4.so.1 to /tmp/velox4j-307006935592892106/lib/liblz4.so.1...
Copying library velox4j-lib/Linux/amd64/libicudata.so.66 to /tmp/velox4j-307006935592892106/lib/libicudata.so.66...
Copying library velox4j-lib/Linux/amd64/libglog.so.0 to /tmp/velox4j-307006935592892106/lib/libglog.so.0...
Copying library velox4j-lib/Linux/amd64/libvelox4j.so to /tmp/velox4j-307006935592892106/lib/libvelox4j.so...
Copying library velox4j-lib/Linux/amd64/libbz2.so.1.0 to /tmp/velox4j-307006935592892106/lib/libbz2.so.1.0...
Copying library velox4j-lib/Linux/amd64/libsnappy.so.1 to /tmp/velox4j-307006935592892106/lib/libsnappy.so.1...
Copying library velox4j-lib/Linux/amd64/libcppkafka.so.0.4.1 to /tmp/velox4j-307006935592892106/lib/libcppkafka.so.0.4.1...
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0319 14:33:22.678424 2944230 JniLoader.cc:29] Initializing Velox4J...
I0319 14:33:22.679422 2944230 JniLoader.cc:43] Velox4J initialized.
All required libraries were successfully loaded.
I0319 14:33:22.850020 2944230 HiveConnector.cpp:56] Hive connector connector-hive created with maximum of 20000 cached file handles with expiration of 0ms.
E0319 14:33:41.576027 2945031 MemoryManager.cc:264] [Velox4J MemoryManager DTOR] Memory leak found on Velox memory pool: Memory Pool[Decoding Memory Pool LEAF root[root] parent[root] MALLOC track-usage thread-safe]. Please make sure your code released all opened resources already.
E0319 14:33:41.576107 2945031 MemoryManager.cc:264] [Velox4J MemoryManager DTOR] Memory leak found on Velox memory pool: Memory Pool[root AGGREGATE root[root] parent[null] MALLOC track-usage thread-safe]. Please make sure your code released all opened resources already.
E0319 14:33:41.576125 2945031 MemoryPool.cpp:461] [MEM] Memory leak (Used memory): Memory Pool[Decoding Memory Pool LEAF root[root] parent[root] MALLOC track-usage thread-safe]
E0319 14:33:41.576150 2945031 Exceptions.h:66] Line: ~/velox4j/src/main/cpp/main/velox4j/memory/MemoryManager.cc:92, Function:removePool, Expression: pool->reservedBytes() == 0 (1048576 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
W0319 14:33:41.576576 2945031 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: facebook::velox::VeloxRuntimeError
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (1048576 vs. 0)
Retriable: False
Expression: pool->reservedBytes() == 0
Function: removePool
File: ~/velox4j/src/main/cpp/main/velox4j/memory/MemoryManager.cc
Line: 92
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN7velox4j12_GLOBAL__N_120ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
# 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
# 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
# 6 _ZN7velox4j13MemoryManager11tryDestructEv
# 7 _ZN7velox4j13MemoryManagerD2Ev
# 8 _ZNSt10_HashtableIjSt4pairIKjSt10shared_ptrIvEESaIS4_ENSt8__detail10_Select1stESt8equal_toIjESt4hashIjENS6_18_Mod_range_hashingENS6_20_Default_ranged_hashENS6_20_Prime_rehash_policyENS6_17_Hashtable_traitsILb0ELb0ELb1EEEE8_M_eraseEmPNS6_15_Hash_node_baseEPNS6_10_Hash_nodeIS4_Lb0EEE
# 9 _ZN7velox4j11ResourceMapISt10shared_ptrIvEE5eraseEj
# 10 _ZN7velox4j11ObjectStore15releaseInternalEj
# 11 _ZN7velox4j12_GLOBAL__N_116releaseCppObjectEP7JNIEnv_P8_jobjectl
# 12 0x00007f92d87cea0f
# 13 0x00007f92d87c9272
# 14 0x00007f92d87c9272
# 15 0x00007f92d87c9272
# 16 0x00007f92d87c92b7
# 17 0x00007f92d87c9272
# 18 0x00007f92d87c92b7
# 19 0x00007f92d87c9272
# 20 0x00007f92d87c92b7
# 21 0x00007f92d87c9272
# 22 0x00007f92d87c92b7
# 23 0x00007f92d87c9272
# 24 0x00007f92d87c9272
# 25 0x00007f92d87c9272
# 26 0x00007f92d87c92b7
# 27 0x00007f92d87c9272
# 28 0x00007f92d87c92b7
# 29 0x00007f92d87c9272
# 30 0x00007f92d87c9272
# 31 0x00007f92d87c9272
# 32 0x00007f92d87c92b7
# 33 0x00007f92d87bfcc8
# 34 _ZN9JavaCalls11call_helperEP9JavaValueRK12methodHandleP17JavaCallArgumentsP6Thread
# 35 _ZN9JavaCalls12call_virtualEP9JavaValue6HandleP5KlassP6SymbolS6_P6Thread
# 36 _ZL12thread_entryP10JavaThreadP6Thread
# 37 _ZN10JavaThread17thread_main_innerEv
# 38 _ZN6Thread8call_runEv
# 39 _ZL19thread_native_entryP6Thread
# 40 start_thread
# 41 clone
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Reproduce Nexmark q4 with the RocksDB state backend, then start at src/main/cpp/main/velox4j/memory/MemoryManager.cc:92 and follow the removePool teardown shown in the stack trace. Done means the task manager no longer crashes during cleanup and the Velox memory manager reports no leaked or reserved memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.