[VL] Memory leak for ColumnarPartialProject
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
If spark.gluten.sql.columnar.partial.project = true
```shell
velox_ep/velox/exec/Task.cpp:2409, Function:terminate, Expression: Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
E20251229 02:43:05.733094 177 MemoryPool.cpp:460] [MEM] Memory leak (Used memory): Memory Pool[default_leaf LEAF root[root] parent[root] MALLOC track-usage thread-safe]
E20251229 02:43:05.733157 177 Exceptions.h:66] Line: gluten/cpp/velox/memory/VeloxMemoryManager.cc:110, Function:removePool, Expression: pool->reservedBytes() == 0 (1275068416 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (1275068416 vs. 0)
Retriable: False
Expression: pool->reservedBytes() == 0
Function: removePool
File: gluten/cpp/velox/memory/VeloxMemoryManager.cc
Line: 110
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN6gluten12_GLOBAL__N_120ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
# 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
# 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
# 6 _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE24_M_release_last_use_coldEv
# 7 _ZN6gluten18VeloxMemoryManagerD1Ev
# 8 _ZN6gluten18VeloxMemoryManagerD0Ev
# 9 _ZN6gluten13MemoryManager7releaseEPS0_
# 10 Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 11 0x00007f983a1e445c
```
After disabling spark.gluten.sql.columnar.partial.project, the issue disappeared.
### Gluten version
Gluten-1.5
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Reproduce with spark.gluten.sql.columnar.partial.project=true and compare behavior when it is disabled. Start with velox_ep/velox/exec/Task.cpp:2409, MemoryPool.cpp:460, and gluten/cpp/velox/memory/VeloxMemoryManager.cc:110, following ListenableArbitrator::removePool. Done means the ColumnarPartialProject path no longer reports leaked or reserved memory and terminates cleanly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100