oceanbase / oceanbase/seekdb

Fix tenant memory leak during tenant deletion (tenant_id=1002, label="VecIdxAdaptor")

Open
#1,085 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dima type: task
Dominant language
C++
Stars
2.9k
Forks
339
Avg merge
1d 19h
Merged PRs (30d)
52

Description

Environment

  • Version: observer (OceanBase)
  • Revision: 1-48b340822cf15e35fe516e3ec3538b44d8f78086
  • Build Branch: vec_transfer_migrate_460
  • Build Time: Jul 8 2026 17:50:11
  • Build Flags: RelWithDebInfo
  • Build Info: obbuild-release-vec_transfer_migrate_460-900012
  • LIBPALF_VERSION: 1.2.0 (12026040115.al8-28810b72f8b678839f9cc851b11d0ef08d3419c4)

Is this a temporary version provided by RD?
No

Reproducibility:
Not specified

Test Changes:
Not specified

Other environmental anomalies or changes:
No

Description

During the tenant deletion process (specifically tenant_id=1002), a memory leak error is reported. The error occurs when recycling the tenant's memory allocator, with the leaked memory labeled as "VecIdxAdaptor".

Log Information

Error Log:

[2026-07-14 15:58:45.834385] EDIAG [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:749) [3914803][OmtNodeBalancer][T0][Y1787060CE9A1-0006564C40C73AE8-0-0] [lt=6][errcode=0] tenant memory leak!!!(tenant_id=1002, ctx_id=0, ctx_name="DEFAULT_CTX_ID", label="VecIdxAdaptor", backtrace="0x34c81b90 0x10896e6d 0x10896060 0x1086f997 0x10879d25 0x108ae1a3 0x108addfe 0x108ad9dc 0x228bfae9 0x228c02e8 0x2292d0cd 0x22a49201 0x22a4a419 0x22a36022 0x22a2f9b5 0x342fc10c") BACKTRACE:0x34c747eb 0x34a478b2 0x108946c7 0x1089412a 0x10893eff 0x10893ce6 0x10893951 0x10892c30 0x13a1a2be 0x13a72991 0x13a6d7c8 0x13a6cbcd 0x34c5a37f 0x34c6e7b6 0x7fe0d8f3d3fb 0x7fe0d8d93e83

Backtrace:

oceanbase::common::light_backtrace(void**, int, long) at /home/distcc/tmp/./deps/oblib/src/lib/utility/ob_backtrace.cpp:27
oceanbase::lib::ObTenantCtxAllocator::on_alloc(oceanbase::lib::AObject&, oceanbase::lib::ObMemAttr const&, bool) at ??:?
void* oceanbase::lib::ObTenantCtxAllocator::common_realloc<oceanbase::lib::ObjectMgr>(void const*, long, oceanbase::lib::ObMemAttr const&, oceanbase::lib::ObTenantCtxAllocator&, oceanbase::lib::ObjectMgr&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.h:421
oceanbase::lib::ObTenantCtxAllocatorGuard::revert at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_malloc_allocator.h:58
oceanbase::common::ob_malloc(long, oceanbase::lib::ObMemAttr const&) at /home/distcc/tmp/./deps/oblib/src/lib/allocator/ob_malloc.h:30
oceanbase::common::ObFIFOAllocator::alloc_new_normal_page at /home/distcc/tmp/./deps/oblib/src/lib/allocator/ob_fifo_allocator.cpp:311
oceanbase::common::ObFIFOAllocator::try_alloc(long, long) at /home/distcc/tmp/./deps/oblib/src/lib/allocator/ob_fifo_allocator.cpp:339
oceanbase::common::ObFIFOAllocator::inner_alloc_align(long, long) at ??:?
int oceanbase::share::ObPluginVectorIndexAdaptor::init_mem<oceanbase::share::ObVecIdxFrozenData>(oceanbase::share::ObVectorIndexMemDataHandle<oceanbase::share::ObVecIdxFrozenData>&) at /home/distcc/tmp/./src/share/vector_index/ob_plugin_vector_index_adaptor.cpp:665 (discriminator 4)
oceanbase::share::ObPluginVectorIndexAdaptor::init(oceanbase::common::ObString, long, oceanbase::lib::MemoryContext&, unsigned long*) at /home/distcc/tmp/./src/share/vector_index/ob_plugin_vector_index_adaptor.cpp:747 (discriminator 2)
oceanbase::share::ObPluginVectorIndexService::acquire_adapter_guard(oceanbase::share::ObLSID, oceanbase::share::ObVectorIndexAcquireCtx&, oceanbase::share::ObPluginVectorIndexAdapterGuard&, oceanbase::common::ObString*, long) at /home/distcc/tmp/./src/share/vector_index/ob_plugin_vector_index_service.cpp:739 (discriminator 2)
oceanbase::share::ObVecIdxAsyncTaskScheduler::create_or_update_adaptors(oceanbase::storage::ObLS*, oceanbase::share::ObPluginVectorIndexMgr*, oceanbase::common::hash::ObHashMap<oceanbase::share::ObPluginVectorIndexIdentity, oceanbase::share::ObVectorIndexAcquireCtx, oceanbase::common::hash::LatchReadWriteDefendMode, oceanbase::common::hash::hash_func<oceanbase::share::ObPluginVectorIndexIdentity>, oceanbase::common::hash::equal_to<oceanbase::share::ObPluginVectorIndexIdentity>, oceanbase::common::hash::SimpleAllocer<oceanbase::common::hash::ObHashTableNode<oceanbase::common::hash::HashMapPair<oceanbase::share::ObPluginVectorIndexIdentity, oceanbase::share::ObVectorIndexAcquireCtx> >, 40, oceanbase::common::hash::SpinMutexDefendMode, oceanbase::common::hash::DefaultSimpleAllocerAllocator>, oceanbase::common::hash::NormalPointer, oceanbase::common::ObMalloc, 1l>&) at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:1116 (discriminator 6)
oceanbase::share::ObVecIdxAsyncTaskScheduler::execute_adapter_maintenance_for_ls(oceanbase::storage::ObLS*, oceanbase::share::ObPluginVectorIndexMgr*, oceanbase::common::ObIArray<unsigned long> const&, bool) at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:1243 (discriminator 6)
oceanbase::share::ObVecIdxAsyncTaskScheduler::check_and_execute_adapter_maintenance_tasks at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:1313 (discriminator 8)
oceanbase::share::ObVecIdxAsyncTaskScheduler::run_timer_task at /home/distcc/tmp/./src/share/vector_index/ob_vec_idx_async_task_scheduler.cpp:313 (discriminator 2)
oceanbase::common::ObTimerTaskThreadPool::clear_ext_tname at /home/distcc/tmp/./deps/oblib/src/lib/task/ob_timer_service.cpp:157

Additional Log Context:
The logs show the tenant deletion process for tenant_id=1002, including the removal of cgroup directories, cache flushing, and other cleanup steps, culminating in the memory leak error.

Environment Information

Error Environment:
http://:4067/data/1/work3/obdocker.z3.obs1/

Cluster Environment:

  • http://:6471/data/1/work3/obdocker.proxy0/
  • http://:29527/data/1/work3/obdocker.z1.obs0/
  • http://:17827/data/1/work3/obdocker.z1.obs1/
  • http://:12363/data/1/work3/obdocker.z3.obs0/
  • http://:4067/data/1/work3/obdocker.z3.obs1/
  • http://:8767/data/1/work3/obdocker.z2.obs0/
  • http://:34017/data/1/work3/obdocker.z2.obs1/

Direct Local Connection String (requires read permission for run/sql.sock file, execute in observer root directory):

mysql -S run/sql.sock -uroot -Doceanbase -Ac

Other

No additional information provided.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the backtrace locations in src/share/vector_index/ob_plugin_vector_index_adaptor.cpp:665 and :747, then trace the calls through ob_plugin_vector_index_service.cpp:739 and ob_vec_idx_async_task_scheduler.cpp:1116. Reproduce tenant deletion in the provided environment and inspect the VecIdxAdaptor allocation lifecycle; done means tenant recycling no longer reports the described leak.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.