Write operations that immediately follow write operations sometimes cause a disk I/O-error, followed by loss of leadership and high latency
- Dominant language
- C
- Stars
- 4.4k
- Forks
- 254
- PR merge metrics
- No merged PRs in 30d
Description
I am using dqlite (version 1.14) for an internal project and I observed some unexpected behavior in my benchmarks (on localhost): If I rapidly spam write-operations (`INSERT OR REPLACE INTO kv_table (KEY, VALUE) VALUES (?,?);`, using request type 8 of the wire protocol), there are some random latency spikes (see picture) that do not appear, if I wait 1ms between requests. What happens is that these outlier requests return SQLite's "disk I/O error", and retrying the request returns "not leader" for some time. I suspect what happens is that this bug triggers a leader election. The files are on a ramdisk and I cannot reproduce the bug if the files are on an SSD, so the bug is probably timing-related.

Regarding the plot: blue dots are 100 write operations on node 1, red dots are 100 read-operations on node 2 (1st red dot is a leadership transfer to node 2). There were 3 voting nodes in the cluster.
Contributor guide
Research direction
Reproduce the benchmark using request type 8 with rapid INSERT OR REPLACE writes on a ramdisk, then compare it with 1 ms delays and SSD-backed files. Trace the disk I/O error and subsequent leadership loss across the three-node cluster; done means identifying the timing-related cause and preventing the error, election, and latency spike.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, sqlite
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100