Fix tenant memory leak during tenant deletion (tenant_id=1002, label="VecIdxAdaptor")
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 339
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 52
Description
Environment
- Version: observer (OceanBase)
- Revision: 1-48b340822cf15e35fe516e3ec3538b44d8f78086
- Build Branch: vec_transfer_migrate_460
- Build Time: Jul 8 2026 17:50:11
- Build Flags: RelWithDebInfo
- Build Info: obbuild-release-vec_transfer_migrate_460-900012
- LIBPALF_VERSION: 1.2.0 (12026040115.al8-28810b72f8b678839f9cc851b11d0ef08d3419c4)
Is this a temporary version provided by RD?
No
Reproducibility:
Not specified
Test Changes:
Not specified
Other environmental anomalies or changes:
No
Description
During the tenant deletion process (specifically tenant_id=1002), a memory leak error is reported. The error occurs when recycling the tenant's memory allocator, with the leaked memory labeled as "VecIdxAdaptor".
Log Information
Error Log:
[2026-07-14 15:58:45.834385] EDIAG [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:749) [3914803][OmtNodeBalancer][T0][Y1787060CE9A1-0006564C40C73AE8-0-0] [lt=6][errcode=0] tenant memory leak!!!(tenant_id=1002, ctx_id=0, ctx_name="DEFAULT_CTX_ID", label="VecIdxAdaptor", backtrace="0x34c81b90 0x10896e6d 0x10896060 0x1086f997 0x10879d25 0x108ae1a3 0x108addfe 0x108ad9dc 0x228bfae9 0x228c02e8 0x2292d0cd 0x22a49201 0x22a4a419 0x22a36022 0x22a2f9b5 0x342fc10c") BACKTRACE:0x34c747eb 0x34a478b2 0x108946c7 0x1089412a 0x10893eff 0x10893ce6 0x10893951 0x10892c30 0x13a1a2be 0x13a72991 0x13a6d7c8 0x13a6cbcd 0x34c5a37f 0x34c6e7b6 0x7fe0d8f3d3fb 0x7fe0d8d93e83
Backtrace:
oceanbase::common::light_backtrace(void**, int, long) at /home/distcc/tmp/./deps/oblib/src/lib/utility/ob_backtrace.cpp:27
oceanbase::lib::ObTenantCtxAllocator::on_alloc(oceanbase::lib::AObject&, oceanbase::lib::ObMemAttr const&, bool) at ??:?
void* oceanbase::lib::ObTenantCtxAllocator::common_realloc<oceanbase::lib::ObjectMgr>(void const*, long, oceanbase::lib::ObMemAttr const&, oceanbase::lib::ObTenantCtxAllocator&, oceanbase::lib::ObjectMgr&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.h:421
oceanbase::lib::ObTenantCtxAllocatorGuard::revert at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_malloc_allocator.h:58
oceanbase::common::ob_malloc(long, oceanbase::lib::ObMemAttr const&) at /home/distcc/tmp/./deps/oblib/src/lib/allocator/ob_malloc.h:30
oceanbase::common::ObFIFOAllocator::alloc_new_normal_page at /home/distcc/tmp/./deps/oblib/src/lib/allocator/ob_fifo_allocator.cpp:311
oceanbase::common::ObFIFOAllocator::try_alloc(long, long) at /home/distcc/tmp/./deps/oblib/src/lib/allocator/ob_fifo_allocator.cpp:339
oceanbase::common::ObFIFOAllocator::inner_alloc_align(long, long) at ??:?
int oceanbase::share::ObPluginVectorIndexAdaptor::init_mem<oceanbase::share::ObVecIdxFrozenData>(oceanbase::share::ObVectorIndexMemDataHandle<oceanbase::share::ObVecIdxFrozenData>&) at /home/distcc/tmp/./src/share/vector_index/ob_plugin_vector_index_adaptor.cpp:665 (discriminator 4)
oceanbase::share::ObPluginVectorIndexAdaptor::init(oceanbase::common::ObString, long, oceanbase::lib::MemoryContext&, unsigned long*) at /home/distcc/tmp/./src/share/vector_index/ob_plugin_vector_index_adaptor.cpp:747 (discriminator 2)
oceanbase::share::ObPluginVectorIndexService::acquire_adapter_guard(oceanbase::share::ObLSID, oceanbase::share::ObVectorIndexAcquireCtx&, oceanbase::share::ObPluginVectorIndexAdapterGuard&, oceanbase::common::ObString*, long) at /home/distcc/tmp/./src/share/vector_index/ob_plugin_vector_index_service.cpp:739 (discriminator 2)
oceanbase::share::ObVecIdxAsyncTaskScheduler::create_or_update_adaptors(oceanbase::storage::ObLS*, oceanbase::share::ObPluginVectorIndexMgr*, oceanbase::common::hash::ObHashMap<oceanbase::share::ObPluginVectorIndexIdentity, oceanbase::share::ObVectorIndexAcquireCtx, oceanbase::common::hash::LatchReadWriteDefendMode, oceanbase::common::hash::hash_func<oceanbase::share::ObPluginVectorIndexIdentity>, oceanbase::common::hash::equal_to<oceanbase::share::ObPluginVectorIndexIdentity>, oceanbase::common::hash::SimpleAllocer<oceanbase::common::hash::ObHashTableNode<oceanbase::common::hash::HashMapPair<oceanbase::share::ObPluginVectorIndexIdentity, oceanbase::share::ObVectorIndexAcquireCtx> >, 40, oceanbase::common::hash::SpinMutexDefendMode, oceanbase::common::hash::DefaultSimpleAllocerAllocator>, oceanbase::common::hash::NormalPointer, oceanbase::common::ObMalloc, 1l>&) at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:1116 (discriminator 6)
oceanbase::share::ObVecIdxAsyncTaskScheduler::execute_adapter_maintenance_for_ls(oceanbase::storage::ObLS*, oceanbase::share::ObPluginVectorIndexMgr*, oceanbase::common::ObIArray<unsigned long> const&, bool) at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:1243 (discriminator 6)
oceanbase::share::ObVecIdxAsyncTaskScheduler::check_and_execute_adapter_maintenance_tasks at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:1313 (discriminator 8)
oceanbase::share::ObVecIdxAsyncTaskScheduler::run_timer_task at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:313 (discriminator 2)
oceanbase::common::ObTimerTaskThreadPool::clear_ext_tname at /home/distcc/tmp/./deps/oblib/src/lib/task/ob_timer_service.cpp:157
Additional Log Context:
The logs show the tenant deletion process for tenant_id=1002, including the removal of cgroup directories, cache flushing, and other cleanup steps, culminating in the memory leak error.
Environment Information
Error Environment:
http://:4067/data/1/work3/obdocker.z3.obs1/
Cluster Environment:
- http://:6471/data/1/work3/obdocker.proxy0/
- http://:29527/data/1/work3/obdocker.z1.obs0/
- http://:17827/data/1/work3/obdocker.z1.obs1/
- http://:12363/data/1/work3/obdocker.z3.obs0/
- http://:4067/data/1/work3/obdocker.z3.obs1/
- http://:8767/data/1/work3/obdocker.z2.obs0/
- http://:34017/data/1/work3/obdocker.z2.obs1/
Direct Local Connection String (requires read permission for run/sql.sock file, execute in observer root directory):
mysql -S run/sql.sock -uroot -Doceanbase -Ac
Other
No additional information provided.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the backtrace locations in src/share/vector_index/ob_plugin_vector_index_adaptor.cpp:665 and :747, then trace the calls through ob_plugin_vector_index_service.cpp:739 and ob_vec_idx_async_task_scheduler.cpp:1116. Reproduce tenant deletion in the provided environment and inspect the VecIdxAdaptor allocation lifecycle; done means tenant recycling no longer reports the described leak.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100