apache / apache/gluten

[VL] Memory leak for ColumnarPartialProject

Open
#11,336 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

If spark.gluten.sql.columnar.partial.project = true
```shell
velox_ep/velox/exec/Task.cpp:2409, Function:terminate, Expression: Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
E20251229 02:43:05.733094 177 MemoryPool.cpp:460] [MEM] Memory leak (Used memory): Memory Pool[default_leaf LEAF root[root] parent[root] MALLOC track-usage thread-safe]
E20251229 02:43:05.733157 177 Exceptions.h:66] Line: gluten/cpp/velox/memory/VeloxMemoryManager.cc:110, Function:removePool, Expression: pool->reservedBytes() == 0 (1275068416 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (1275068416 vs. 0)
Retriable: False
Expression: pool->reservedBytes() == 0
Function: removePool
File: gluten/cpp/velox/memory/VeloxMemoryManager.cc
Line: 110
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN6gluten12_GLOBAL__N_120ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
# 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
# 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
# 6 _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE24_M_release_last_use_coldEv
# 7 _ZN6gluten18VeloxMemoryManagerD1Ev
# 8 _ZN6gluten18VeloxMemoryManagerD0Ev
# 9 _ZN6gluten13MemoryManager7releaseEPS0_
# 10 Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 11 0x00007f983a1e445c

```

After disabling spark.gluten.sql.columnar.partial.project, the issue disappeared.

### Gluten version

Gluten-1.5

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Reproduce with spark.gluten.sql.columnar.partial.project=true and compare behavior when it is disabled. Start with velox_ep/velox/exec/Task.cpp:2409, MemoryPool.cpp:460, and gluten/cpp/velox/memory/VeloxMemoryManager.cc:110, following ListenableArbitrator::removePool. Done means the ColumnarPartialProject path no longer reports leaked or reserved memory and terminates cleanly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.