oceanbase / oceanbase/miniob

[BUG] dead lock

Open
#238 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
4.4k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Describe the bug
A clear and concise description of what the bug is.

A dead lock scenario:
buffer pool bpA allocate_page and there are no free frames

  1. bpA lock itself
  2. frame manager purge_frames: lock frame manager
  3. It want to purge a frame of another buffer pool(bpB) and try to lock another buffer pool

Another thread:

  1. bpB lock itself
  2. bpB want to lock frame manager

so the dead lock:
thread 1:
bpA locked
frame manager locked
try to lock bpB

thread 2:
bpB locked
try to lock frame manager

Fast Reproduce Steps(Required)
Steps to reproduce the behavior:
create multiple tables. execute a inserting performance test. wait the frame exhaust.

Expected behavior
A clear and concise description of what you expected to happen.
No dead lock.

Actual Behavior
What is the result? picture is allowed

(gdb) thread 2
[Switching to thread 2 (Thread 0x7fd6f860d700 (LWP 295211))]
#0  0x00007fd6ff06351d in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fd6ff06351d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fd6ff05ee1b in _L_lock_812 () from /lib64/libpthread.so.0
#2  0x00007fd6ff05ece8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004dddf0 in __gthread_mutex_lock (__mutex=0x60e000007518)
    at /usr/opt/include/c++/9.4.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749
#4  0x00000000004ddfc0 in std::mutex::lock (this=0x60e000007518) at /usr/opt/include/c++/9.4.0/bits/std_mutex.h:100
#5  0x000000000078b111 in common::Mutex::lock (this=0x60e000007518)
    at /home/xiongliyao.xly/miniob-group2/miniob/deps/common/lang/mutex.cpp:281
#6  0x00000000004e5681 in std::scoped_lock<common::Mutex>::scoped_lock (this=0x7fd6f85fbde0, __m=...)
    at /usr/opt/include/c++/9.4.0/mutex:609
#7  0x00000000004c572b in DiskBufferPool::flush_page (this=0x60e0000074a0, frame=...)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/buffer/disk_buffer_pool.cpp:504
#8  0x00000000004da398 in BufferPoolManager::flush_page (this=0x615000000300, frame=...)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/buffer/disk_buffer_pool.cpp:769
#9  0x00000000004ca789 in DiskBufferPool::<lambda(Frame*)>::operator()(Frame *) const (__closure=0x7fd6f85ff310, 
    frame=0x7fd6faa8cc70) at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/buffer/disk_buffer_pool.cpp:572
#10 0x00000000004dcfda in std::_Function_handler<RC(Frame*), DiskBufferPool::allocate_frame(PageNum, Frame**)::<lambda(Frame*)> >:---Type <return> to continue, or q <return> to quit---
:_M_invoke(const std::_Any_data &, <unknown type in /home/xiongliyao.xly/miniob-group2/miniob/build_debug/bin/observer, CU 0x26d1e0, DIE 0x2b8163>) (__functor=..., 
    __args#0=<unknown type in /home/xiongliyao.xly/miniob-group2/miniob/build_debug/bin/observer, CU 0x26d1e0, DIE 0x2b8163>)
    at /usr/opt/include/c++/9.4.0/bits/std_function.h:285
#11 0x00000000004e3549 in std::function<RC (Frame*)>::operator()(Frame*) const (this=0x7fd6f85ff310, __args#0=0x7fd6faa8cc70)
    at /usr/opt/include/c++/9.4.0/bits/std_function.h:688
#12 0x00000000004a8195 in BPFrameManager::purge_frames(int, std::function<RC (Frame*)>) (this=0x615000000300, count=1, 
    purger=...) at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/buffer/disk_buffer_pool.cpp:91
#13 0x00000000004cbf14 in DiskBufferPool::allocate_frame (this=0x60e0000100a0, page_num=407, buffer=0x7fd6f85ff940)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/buffer/disk_buffer_pool.cpp:589
#14 0x00000000004bcda4 in DiskBufferPool::allocate_page (this=0x60e0000100a0, frame=0x7fd6f8601bf0)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/buffer/disk_buffer_pool.cpp:378
#15 0x000000000075e852 in LatchMemo::allocate_page (this=0x7fd6f86029a0, fra me=@0x7fd6f8601bf0: 0x0)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/trx/latch_memo.cpp:59
#16 0x000000000074e234 in BplusTreeHandler::split<LeafIndexNodeHandler> (this=0x611000019fc0, latch_memo=..., 
    frame=0x7fd6f9939a58, new_frame=@0x7fd6f8601bf0: 0x0)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/index/bplus_tree.cpp:1281
---Type <return> to continue, or q <return> to quit---
#17 0x000000000072daba in BplusTreeHandler::insert_entry_into_leaf_node (this=0x611000019fc0, latch_memo=..., 
    frame=0x7fd6f9939a58, key=0x61b000000548 "^", rid=0x7fd6f8605660)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/index/bplus_tree.cpp:1156
#18 0x0000000000737860 in BplusTreeHandler::insert_entry (this=0x611000019fc0, user_key=0x6100024d2044 "^", rid=0x7fd6f8605660)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/index/bplus_tree.cpp:1380
#19 0x00000000006b34af in BplusTreeIndex::insert_entry (this=0x611000019f40, record=0x6100024d2040 "\370R\243\316^", 
    rid=0x7fd6f8605660) at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/index/bplus_tree_index.cpp:92
#20 0x000000000059ac24 in Table::insert_entry_of_indexes (this=0x60f000007840, record=0x6100024d2040 "\370R\243\316^", rid=...)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/table/table.cpp:451
#21 0x0000000000585c2a in Table::insert_record (this=0x60f000007840, record=...)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/table/table.cpp:190
#22 0x00000000005c03af in VacuousTrx::insert_record (this=0x60200000ff90, table=0x60f000007840, record=...)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/storage/trx/vacuous_trx.cpp:56
#23 0x000000000062e76e in InsertPhysicalOperator::open (this=0x6060026c6600, trx=0x60200000ff90)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/sql/operator/insert_physical_operator.cpp:35
#24 0x0000000000457c8c in SqlResult::open (this=0x6110040363c0)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/sql/executor/sql_result.cpp:37
---Type <return> to continue, or q <return> to quit---
#25 0x00000000005d3122 in MysqlCommunicator::write_result (this=0x611000003240, event=0x611004036380, 
    need_disconnect=@0x7fd6f860a2b0: true)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/net/mysql_communicator.cpp:765
#26 0x000000000044b6e2 in SessionStage::handle_request (this=0x6120000001c0, event=0x611004036380)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/session/session_stage.cpp:111
#27 0x000000000044943d in SessionStage::handle_event (this=0x6120000001c0, event=0x611004036380)
    at /home/xiongliyao.xly/miniob-group2/miniob/src/observer/session/session_stage.cpp:82
#28 0x000000000083836d in common::Threadpool::execute_stage (run_stage=0x6120000001c0, pool=0x616000000c80)
    at /home/xiongliyao.xly/miniob-group2/miniob/deps/common/seda/thread_pool.cpp:423
#29 0x0000000000837fcd in common::Threadpool::run_thread (pool_ptr=0x616000000c80)
    at /home/xiongliyao.xly/miniob-group2/miniob/deps/common/seda/thread_pool.cpp:368
#30 0x00007fd6ff05ce25 in start_thread () from /lib64/libpthread.so.0
#31 0x00007fd6fe28fbad in clone () from /lib64/libc.so.6

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/observer/storage/buffer/disk_buffer_pool.cpp, especially BPFrameManager::purge_frames, DiskBufferPool::allocate_frame, and flush_page, using the reported stack and lock sequence to trace buffer-pool and frame-manager ownership. Reproduce by creating multiple tables and running inserts until frames are exhausted; done means the workload no longer deadlocks.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.