canonical / canonical/dqlite

Write operations that immediately follow write operations sometimes cause a disk I/O-error, followed by loss of leadership and high latency

Open
#522 3 comments 0 reactions 0 assignees View on GitHub
Bug Incomplete
Dominant language
C
Stars
4.4k
Forks
254
PR merge metrics
No merged PRs in 30d

Description

I am using dqlite (version 1.14) for an internal project and I observed some unexpected behavior in my benchmarks (on localhost): If I rapidly spam write-operations (`INSERT OR REPLACE INTO kv_table (KEY, VALUE) VALUES (?,?);`, using request type 8 of the wire protocol), there are some random latency spikes (see picture) that do not appear, if I wait 1ms between requests. What happens is that these outlier requests return SQLite's "disk I/O error", and retrying the request returns "not leader" for some time. I suspect what happens is that this bug triggers a leader election. The files are on a ramdisk and I cannot reproduce the bug if the files are on an SSD, so the bug is probably timing-related.

![dqlite-io-erros](https://github.com/canonical/dqlite/assets/7428549/17ee9790-5b89-4e7a-8ebb-bdac417a11e7)
Regarding the plot: blue dots are 100 write operations on node 1, red dots are 100 read-operations on node 2 (1st red dot is a leadership transfer to node 2). There were 3 voting nodes in the cluster.

Contributor guide

Open the contributing guide

Research direction

Reproduce the benchmark using request type 8 with rapid INSERT OR REPLACE writes on a ramdisk, then compare it with 1 ms delays and SSD-backed files. Trace the disk I/O error and subsequent leadership loss across the three-node cluster; done means identifying the timing-related cause and preventing the error, election, and latency spike.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, sqlite
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.