[Bug]: KVCacheBlock and its radix tree LookupNode own each other with shared_ptr, so blocks are never freed

System Info

• Container: nvcr.io/nvidia/tensorrt-llm/devel:1.3.0rc26
• Commit: 63d217f2 on main (version 1.3.0rc27)
• GPU: NVIDIA B300 SXM6, compute capability 10.3
• CUDA 13.2, GCC 13

Built with the sanitizer option the repository already ships:

python3 scripts/build_wheel.py --skip_building_wheel --skip-stubs \
  --cuda_architectures 103-real --build_type RelWithDebInfo --fast_build \
  -D "SANITIZE=address" -D "BUILD_TESTS=ON"

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

ASAN_OPTIONS=detect_leaks=1:protect_shadow_gap=0:print_stacktrace=1:fast_unwind_on_malloc=0 \
  ./cpp/build_RelWithDebInfo/tests/unit_tests/batch_manager/kvCacheManagerTest

Expected behavior

The KV cache block graph is released when the manager goes away.

actual behavior

SUMMARY: AddressSanitizer: 564624 byte(s) leaked in 3892 allocation(s).

2031 leak blocks in total. 188 of them are allocated through this repository's own frames,
all at the same place:

#0 operator new
#1 std::vector<tensorrt_llm::runtime::UniqueToken>::operator=
#2 tensorrt_llm::batch_manager::kv_cache_manager::BlockKey::operator=
#3 tensorrt_llm::batch_manager::kv_cache_manager::KVCacheBlock::setBlockKey
     cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp:264
#4 WindowBlockManager::onboardAndAllocateBlocks
#5 WindowBlockManager::addSequenceBatch
#6 BlockManager::addSequenceBatch
#7 KVCacheManager::addSequenceBatch
#8 testKVCacheManagerLinearAttention_BlockCopying
     cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp:8010

A "direct leak" means nothing points at the memory any more, so the blocks holding those
BlockKeys were not destroyed.

Why the blocks are not destroyed

The block and its lookup node own each other, and both sides are strong:

// cpp/include/tensorrt_llm/batch_manager/radixBlockTree.h:44,54
using BlockPtr      = std::shared_ptr<kv_cache_manager::KVCacheBlock>;
using LookupNodePtr = std::shared_ptr<LookupNode>;

// same file, UnifiedBlockTree stores the block as the trie Value
class UnifiedBlockTree : public templated_trie::Trie<BlockKey, BlockKeyHasher, int,
                                                     std::hash<int>, BlockPtr, true>

// cpp/include/tensorrt_llm/batch_manager/kvCacheManager.h:563
radix_block_tree::LookupNodePtr mLookupNode;

grep -c weak_ptr returns 0 for both radixBlockTree.h and kvCacheManager.h, so nothing
breaks the cycle by construction. The comment on insertBlock calls the addNextBlock
path "bidirectional wiring", which is the cycle.

KVCacheBlock::detachFromLookupNode() does break it, but it is called from the eviction and
pruning paths only (kvCacheManager.cpp:414, kvCacheManager.cpp:473). There is no
~KVCacheBlock, and teardown of the manager does not walk the tree to detach. Blocks still
attached when the manager dies therefore keep their node alive and vice versa.

What I have not measured

This is an at-exit leak in a unit test. I have not measured whether the same cycle makes
memory grow during serving. If every block that leaves the tree goes through
detachFromLookupNode, steady-state serving may be unaffected and only teardown leaks. I
could not separate those two cases from a single test run, so I am reporting what I can show
rather than claiming a runtime leak.

What the measurement does establish is that destroying the manager does not free its blocks,
and that the ownership graph has no weak edge that would let it.

additional notes

Found while running the repository's own SANITIZE=address build, which appears not to be
exercised by CI. Note that such a build needs ASAN_OPTIONS=protect_shadow_gap=0, otherwise
CUDA context creation fails with cudaErrorMemoryAllocation and most tests never reach
their body.

Start with cpp/include/tensorrt_llm/batch_manager/radixBlockTree.h and kvCacheManager.h to trace the shared ownership between KVCacheBlock and LookupNode, then inspect detachFromLookupNode and the teardown paths in cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp. Rebuild with the shown SANITIZE=address options and run cpp/build_RelWithDebInfo/tests/unit_tests/batch_manager/kvCacheManagerTest; done means manager destruction no longer reports the demonstrated leaks while the test passes.