apache / apache/gluten

ArrowNativeMemoryPool memory leak

Open
#5,864 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

From `TestOperator` [github action test ](https://github.com/apache/incubator-gluten/actions/runs/9220817939/job/25371831381?pr=5862)
```
E20240524 09:48:03.909570 7753 Exceptions.h:67] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1858, Function:terminate, Expression: Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
24/05/24 09:48:03 WARN ArrowNativeMemoryPool: Arrow pool still reserved non-zero bytes, which may cause memory leak, size: 128.0 B.
/__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/src/arrow_ep/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: Java Exception: java.lang.IncompatibleClassChangeError

/tmp/jnilib-7232572254174836976.tmp(+0x12183e8)[0x7f5da665e3e8]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow4util8ArrowLogD1Ev+0xed)[0x7f5da665e56d]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow7dataset3jni31ReservationListenableMemoryPool4FreeEPhll+0x45d)[0x7f5da5c5409d]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10PoolBufferD0Ev+0x47)[0x7f5da68ea8f7]
/tmp/jnilib-7232572254174836976.tmp(_ZNSt6vectorISt10shared_ptrIN5arrow6BufferEESaIS3_EED1Ev+0x6b)[0x7f5da5d681fb]
/tmp/jnilib-7232572254174836976.tmp(_ZNSt23_Sp_counted_ptr_inplaceIN5arrow9ArrayDataESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0xda)[0x7f5da5e75e0a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow17SimpleRecordBatchD1Ev+0x116)[0x7f5da67943d6]
/tmp/jnilib-7232572254174836976.tmp(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x48)[0x7f5da5c4d758]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_11RecordBatchEEE20WrapResultOnComplete8CallbackINSC_14ThenOnCompleteIZNS_23DefaultIfEmptyGeneratorISB_EclEvEUt_NSC_17PassthruOnFailureISI_EEEEEEED0Ev+0x2e)[0x7f5da5c9e6ae]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x3a4)[0x7f5da6677124]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_11RecordBatchEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da5cbec6a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_11RecordBatchEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISC_SC_Lb0ELb0EEEEEE6invokeES4_+0x56)[0x7f5da5cbf846]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt8functionIFNS8_ISt10shared_ptrINS_11RecordBatchEEEEvEEE20WrapResultOnComplete8CallbackINSG_14ThenOnCompleteIZNS_20FutureFirstGeneratorISC_EclEvEUlRKSF_E_NSG_17PassthruOnFailureISO_EEEEEEE6invokeES4_+0xde)[0x7f5da5cbfd8e]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(+0x83c23b)[0x7f5da5c8223b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(+0x82d04e)[0x7f5da5c7304e]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_3csv15StreamingReaderEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISD_SD_Lb0ELb0EEEEEE6invokeES4_+0x164)[0x7f5da5c9fe34]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(+0x119ba3b)[0x7f5da65e1a3b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureINS_8internal5EmptyEE12MarkFinishedIS2_vEEvNS_6StatusE+0x94)[0x7f5da5cad6e4]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureINS0_5EmptyEE21WrapStatusyOnComplete8CallbackINS_6detail16MarkNextFinishedISA_SA_Lb1ELb1EEEEEE6invokeES4_+0x28)[0x7f5da5cad848]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
/tmp/jnilib-7232572254174836976.tmp(_ZNK5arrow6FutureINS_8internal5EmptyEE11AddCallbackINS_6detail16MarkNextFinishedIS3_S3_Lb1ELb1EEENS3_21WrapStatusyOnComplete8CallbackIS7_EEEEvT_NS_15CallbackOptionsE+0x5b)[0x7f5da66253bb]
/tmp/jnilib-7232572254174836976.tmp(+0x11c8543)[0x7f5da660e543]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_6BufferEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da6029c4a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_6BufferEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISC_SC_Lb0ELb0EEEEEE6invokeES4_+0x56)[0x7f5da602e706]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_6BufferEEE20WrapResultOnComplete8CallbackINSC_14ThenOnCompleteIZNS_21TransformingGeneratorISB_SB_E26TransformingGeneratorStateclEvEUlRKSB_E_NSC_17PassthruOnFailureISL_EEEEEEE6invokeES4_+0x99)[0x7f5da6623e29]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_6BufferEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da6029c4a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvvEE6FnImplIZZNS0_8Executor10DoTransferISt10shared_ptrINS_6BufferEENS_6FutureIS9_EENS_6ResultIS9_EEEENSA_IT_EESF_bENUlRKSD_E0_clESH_EUlvE_E6invokeEv+0x52)[0x7f5da602e5e2]
/tmp/jnilib-7232572254174836976.tmp(+0x1210abd)[0x7f5da6656abd]
```

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the TestOperator failure from the linked GitHub Actions run. Inspect velox/exec/Task.cpp:1858 and arrow_ep/src/arrow_ep/java/dataset/src/main/cpp/jni_util.cc:79, following the ArrowNativeMemoryPool warning and Java exception. Done means the failing test no longer reports reserved bytes or the incompatible-class-change error.

Written by the indexing model from the issue text.

Assessment

Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.