SQLite in StorageServer deadlocked after the node was disconnected and resumed.
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
My 3-node 3-duplicates fdb cluster.
version:7.1.27
The operation steps are: Node A is disconnected from the network for 10 minutes and restored. During the data moving, Node B is disconnected from the network for ten minutes and then restored. The storageServer of Node B has an infinite loop. The result of pstack is as follows:

The backtrace result in the printed Net2RunLoopTrace is as follows:

I gdb went in and found that I couldn't get a lock from SQLite。I gdb went in and found that I couldn't get a lock from SQLite. Then it tried again indefinitely.
the code is https://github.com/apple/foundationdb/blob/main/contrib/sqlite/sqlite3.amalgamation.c#L37717

the rc is SQLITE_BUSY
the lockIdx is 4 and the n is 4
Contributor guide
Research direction
Reproduce the 3-node, 3-duplicate recovery sequence with Node A and then Node B disconnected for ten minutes during data movement. Start with the StorageServer backtrace and contrib/sqlite/sqlite3.amalgamation.c around line 37717, focusing on the SQLITE_BUSY lockIdx 4 retry; done means the StorageServer resumes without an infinite loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sqlite
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100