apple / apple/foundationdb

SQLite in StorageServer deadlocked after the node was disconnected and resumed.

Open
#11,578 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

My 3-node 3-duplicates fdb cluster.
version:7.1.27
The operation steps are: Node A is disconnected from the network for 10 minutes and restored. During the data moving, Node B is disconnected from the network for ten minutes and then restored. The storageServer of Node B has an infinite loop. The result of pstack is as follows:
![pstack](https://github.com/user-attachments/assets/2423c4ba-9e9c-47e7-8c3c-6f05cd95cf0e)

The backtrace result in the printed Net2RunLoopTrace is as follows:
![20240815-172850](https://github.com/user-attachments/assets/04a8713c-60dc-4ebd-8be4-a429e2ba7e58)

I gdb went in and found that I couldn't get a lock from SQLite。I gdb went in and found that I couldn't get a lock from SQLite. Then it tried again indefinitely.
the code is https://github.com/apple/foundationdb/blob/main/contrib/sqlite/sqlite3.amalgamation.c#L37717
![screenshot-20240815-173116](https://github.com/user-attachments/assets/b8016ef5-3386-46c4-bb37-ea56d7ab4ee9)

the rc is SQLITE_BUSY
the lockIdx is 4 and the n is 4

Contributor guide

Open the contributing guide

Research direction

Reproduce the 3-node, 3-duplicate recovery sequence with Node A and then Node B disconnected for ten minutes during data movement. Start with the StorageServer backtrace and contrib/sqlite/sqlite3.amalgamation.c around line 37717, focusing on the SQLITE_BUSY lockIdx 4 retry; done means the StorageServer resumes without an infinite loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sqlite
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.