Use finer-grained locks to enhance reference table write throughput
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
We currently use shard-level locks to serialise writes to replicated/reference tables, only allowing inserts to run concurrently. In the case of a reference table, shard-level is the same as table-level. This significantly limits write concurrency and thus overall throughput.
For updates/deletes/inserts that are known not to conflict because they have disjoint primary key filters, we could be using more fine-grained locks on primary key values or ranges of primary keys. A thing to watch out for is updates to the primary key value (should perhaps still use a global lock).
Contributor guide
Research direction
No files, tests, or entry points are named. Start by tracing shard-level locking for replicated/reference table writes, then evaluate primary-key or primary-key-range locks for disjoint inserts, updates, and deletes, including the proposed global-lock behavior for primary-key updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, postgresql
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100