citusdata / citusdata/citus

Use finer-grained locks to enhance reference table write throughput

Open
#2,302 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We currently use shard-level locks to serialise writes to replicated/reference tables, only allowing inserts to run concurrently. In the case of a reference table, shard-level is the same as table-level. This significantly limits write concurrency and thus overall throughput.

For updates/deletes/inserts that are known not to conflict because they have disjoint primary key filters, we could be using more fine-grained locks on primary key values or ranges of primary keys. A thing to watch out for is updates to the primary key value (should perhaps still use a global lock).

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing shard-level locking for replicated/reference table writes, then evaluate primary-key or primary-key-range locks for disjoint inserts, updates, and deletes, including the proposed global-lock behavior for primary-key updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.