cockroachdb / cockroachdb/pebble

db: ingest during WAL failover can prevent visibility of later writes

Open
#5,330 0 comments 0 reactions 0 assignees View on GitHub
A-storage T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

(The probability is low, but documenting this for completeness, since we speculated about it https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1757338160400119?thread_ts=1757110210.361039&cid=CAC6K3SLU in a recent escalation, where this did NOT happen https://github.com/cockroachlabs/support/issues/3426)

Say writes to the primary directory are blocked, and WAL failover is configured. Writes will continue to commit successfully. However, if there is an ingest, that does not overlap with the memtable, and therefore does not become a flushable-ingest, it will grab a seqnum and then proceed to attempt to modify the manifest. Since the manifest write will block, normal batch writes that are at a later seqnum cannot become visible.

One solution to this would be convert all ingests while WAL failover is active into flushable ingests. The risk is creation of many small memtables.

One could reduce that risk at the higher layer e.g. CockroachDB can be made aware of WAL failover and can pause admitting operations that require ingest. The difficulty is that such admission happens on the leaseholder of a range, while the WAL failover may be on a follower, which has to ingest once the AddSStable has entered the raft log.

Jira issue: PEBBLE-1184

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.